ansible.it.md

Ansible

Questo documento descrive come l'infrastruttura usa Ansible, come lo si usa da utente, come lo si estende come servizio in sviluppo.

Caratteristiche

Il nostro kit di strumenti è implementato come set di pkugin e ruoli Ansible, significa che è integrato nelle tue proprie configurazioni Ansible. Questi plugin e ruoli aumentano le funzionalità di Ansible in diversi modi utili:

GPG integrata

Puoi usare GPG per cifrate i file contenenti le configurazioni dei gruppi e gli host. Such files must have a .yml.gpg extension. Ansible will then decrypt them at runtime (use of gpg-agent is advised). Same holds for the ansible-vault password file.

Useful when you're using GPG to manage the root trust repository.

This functionality is implemented by the gpg_vars plugin.

Automated credentials management

Service-level credentials are automatically managed by the toolkit and are encrypted with ansible-vault. This includes an internal X509 PKI for TLS service authentication, a SSH PKI for hosts, and application credentials defined in passwords.yml. See the Credentials section of the configuration reference for specific details.

All autogenerated credentials are stored in the credentials_dir specified in the top-level Float configuration, which will usually point at a separate git repository (or a temporary directory for test environments).

Integration of services and Ansible roles

The toolkit defines Ansible host groups for each service, to make it easy to customize services with Ansible roles. For instance, suppose that a service foo is defined in services.yml, and you have created your own foo Ansible role to go with it (with some foo-specific host setup). You can then tie the two together in your playbook by making use of the foo host group:

- hosts: foo
  roles:
    - foo

Usage

The toolkit lets you define container-based services, which may not require any configuration beyond the service definition, as well as Ansible-based services. It does so primarily by taking over the Ansible inventory (see the configuration reference for details). It is expected that you will have your own Ansible configuration, with service-specific roles and configuration, extending the base roles and making use of the toolkit's features.

So, to use the toolkit, you will have to include it from your own Ansible configuration, and specifying the inventory and service configuration in our own format.

There are some (minimal) requirements on how your Ansible environment should be set up for this to work:

• you must have a group_vars/all directory (this is where we'll write the autogenerated application credentials)
• you must include playbooks/all.yml from the toolkit source directory at the beginning of your playbook
• you must use the float wrapper instead of running ansible-playbook directly (it helps setting up the command line)

Ansible environment setup how-to

Let's walk through creating an example Ansible configuration for your project.

First, check out the base float repository somewhere. We'll store everything related to this project below a top-level directory called ~/myproject. We'll put the float repository in the float subdirectory.

$ mkdir ~/myproject
$ git clone ... ~/myproject/float

Let's create the directory with our own Ansible configuration in the ansible subdirectory:

$ mkdir ~/myproject/ansible

And put a top-level Ansible configuration file (ansible.cfg) in there that refers to the toolkit repository location:

$ cat >ansible.cfg <<EOF
[defaults]
roles_path = ../float/roles:./roles
inventory_plugins = ../float/plugins/inventory
action_plugins = ../float/plugins/action
vars_plugins = ../float/plugins/vars
force_handlers = True

[inventory]
enable_plugins = float
EOF

This will look for plugins and base roles in ~/myproject/float, and it will load our own Ansible roles and config from ~/myproject/ansible.

The force_handlers option is important because float controls system status via handlers, and they should run even in case of errors.

We're going to need a place to store global configuration, float requires to have a group_vars/all directory anyway, so we can use that and put some global variables in ~/myproject/ansible/group_vars/all/config.yml:

$ mkdir -p ~/myproject/ansible/group_vars/all
$ cat > ~/myproject/ansible/group_vars/all/config.yml <<EOF
---
domain: internal.myproject.org
domain_public:
  - myproject.org
EOF

Then you can create the main configuration file (float.yml), the host inventory (hosts.yml), and the service definition (services.yml). Check out the configuration reference for details.

Finally, we are going to set up a basic playbook in site.yml that will just run all the playbooks in the main repository:

---
- import_playbook: ../float/playbooks/all.yml

Now you can create your own service-specific Ansible configuration and roles based on this skeleton.

Running playbooks

The float wrapper makes some necessary fixes to the environment and invokes ansible-playbook with the same command-line arguments, so you should use its run command whenever you would use ansible-playbook.

The ansible-vault setup is mandatory, so you are going to have to pass the location of the ansible-vault encryption passphrase to Ansible via the environment variable ANSIBLE_VAULT_PASSWORD_FILE.

With respect to the previous example:

$ echo secret > vault-pw
$ ANSIBLE_VAULT_PASSWORD_FILE=vault-pw \
    ../float/float run --config=config.yml site.yml

Initialize the permanent credentials

Before you can run Ansible to set up the services in your config, there is one more step that needs to be done. In order to bootstrap the internal PKIs, and generate the application credentials (which are also valid forever, or until revoked of course), you need to invoke the playbook in playbooks/init-credentials.yml:

$ ANSIBLE_VAULT_PASSWORD_FILE=vault-pw \
    ../float/float run --config=config.yml init-credentials

This will write a bunch of files in your credentials_dir, including the private keys for the various PKIs (X509, SSH, etc), and a secrets.yml file containing the autogenerated application credentials.

These files are of course to be kept private when setting up a production environment.

Credentials

The system uses two major types of credentials:

• managed credentials for accounts, services, etc - these can be autogenerated automatically, based on the top-level description in passwords.yml. They are stored encrypted with ansible-vault.
• root credentials, that need to be provided externally, including for instance the ansible-vault password used for managed credentials, and other third-party secrets (like credentials for a private Docker registry, etc).

All services that require per-host secrets, such as SSH and the internal X509 PKI, manage those secrets directly on the hosts themselves, renewing them when necessary. Those secrets are not stored in a central location.

This means that for normal usage (i.e. except when new credentials are added), the credentials repository is read-only, which makes it easier to integrate deployment with CI systems.

The expectation is that, for production environments, it will be saved in private git repositories. Temporary setups such as test environments, on the other hand, need no persistence as managed credentials can simply be re-created every time.

Implementation details

These are details of how parts of the infrastructure is implemented, useful if you want to understand how a service is deployed, or how to write a new one.

Scheduler

The float service scheduler is implemented as an Ansible dynamic inventory plugin. All it does is set a number of global and host variables for the float Ansible roles to use. It's useful to know what these are, in case you want to write your own Ansible roles.

Variables

The float service scheduler sets a large number of host variables and global configuration parameters.

The following global variables are defined:

• services holds all the service metadata, in a dictionary indexed by service name;

Other variables are defined in hostvars and are different on every host depending on the service assignments:

• float_enable_<service> for each service that evaluates to true on the hosts assigned to the service (note: dashes in the service name are converted to underscores)
• float_instance_index_<service> is the progressive index of the current instance of the service (0-based).
• float_enabled_services contains the list of enabled services on this host
• float_disabled_services contains the list of disabled services on this host
• float_enabled_containers contains a list of dictionaries describing the containers that should be active on this host. The dictionaries have the following attributes:
  • service is the service metadata
  • container is the container metadata
• float_<service>_is_master is true on the host where the master instance is scheduled, and false elsewhere. This variable is only defined for services using static master election (i.e. where master_election is true in the service metadata)

Groups

The scheduler also defines new dynamic Ansible groups based on service assignments:

• For each service, create a host group named after the service, whose members are the hosts assigned to the service.
• For each network overlay defined in the inventory, create a host group named overlay-<name> whose members are the hosts on that overlay.

Float depends on the frontend group being defined by the user.

Extending the built-in services

Most configurable aspects of the built-in services (monitoring, HTTP routing, etc) are parameterized in services.yml. But there are limits to what makes sense to put there, to avoid unbounded complexity in the service definition spec. Examples could be:

• new monitoring configuration, such as probers or other custom config
• custom HTTP configuration
• custom DNS zones

In those cases, the simplest solution is to create a separate Ansible role to be run on the frontend nodes and install whatever configuration files you need. Unfortunately, due to Ansible not having "global" handlers, you are then going to have to provide your own service reload handlers.

The following chapter shows an example in detail: adding a custom DNS zone to the built-in DNS service.

Custom DNS zone

If you want to set up a custom DNS zone, one way to do so is with a dedicated Ansible role (to be run on hosts in the frontend group) that installs your desired zonetool configuration.

Let's walk through a complete example: suppose we have a service myservice that should serve HTTP requests for the myservice.org domain. This doesn't match the service_name.domain scheme that is expected for services described in services.yml, so float won't automatically generate its DNS configuration.

What we need to do is set up the myservice.org DNS zone ourselves, and then tell float to associate that domain to the myservice service.

First, we create a new Ansible role that we are going to call myservice-dns, so in the root of your Ansible config:

$ mkdir -p roles/myservice-dns/{handlers,tasks,templates}

The only task in the role should install a zonetool DNS configuration file into /etc/dns/manual, so in roles/myservice-dns/tasks/main.yml we'll have:

---

- name: Install myservice DNS configuration
  template:
    src: myservice.yml.j2
    dest: /etc/dns/manual/myservice.yml
  notify: reload DNS

The DNS configuration in our case is very simple and just points "www" and the top-level domain at the frontends. We do so by extending the @base zone template defined by float. The contents of roles/myservice-dns/templates/myservice.yml.j2 should be:

---

myservice.org:
  EXTENDS: "@base"
  www: CNAME www.{{ domain_public[0] }}.

This points the www domain at the frontends via a CNAME (all the domain_public DNS zones are already autogenerated by float). We could have just as easily used A records but this is simpler and works with both IPv4 and IPv6.

Finally, we need a handler to reload the updated DNS configuration, which goes in roles/myservice-dns/handlers/main.yml and runs a shell command to update zonetool:

---

- listen: reload DNS
  shell: "/usr/sbin/update-dns && rndc reload"

With the above we have a complete Ansible role that configures DNS for the myservice.org domain. We need to tell Ansible that this role needs to run on the hosts in the frontend group, so in your playbook you should have:

- hosts: frontend
  roles:
    - myservice-dns

And to complete our configuration, the service description for myservice should have a public_endpoint directive including the domain, so that the float HTTP router knows where to send the requests:

myservice:
  ...
  public_endpoints:
    - name: myservice
      domains:
        - www.myservice.org
        - myservice.org
      port: ...