Introduction

This article describes the configuration management tool Ansible and some of its basic concepts. Then it discusses how to test Ansible code. Refer to the version chart below. It assumes familiarity with the concept of configuration management, programming, and testing.

What is Ansible?

Ansible is a configuration management tool which can be used to manage software installations. Configuration management helps us to ensure that software installations are documented, consistent, and repeatable. Let’s say you want to copy the files for your web application to a server and make sure that its runtime is installed (such as Ruby or .NET Core). While there are many who might deploy manually, it is convenient to be able to do this in an automated way triggered by an event such as an update to source code. Ansible is just one of many tools that can help with this automation and is often compared to alternatives such as Chef, Puppet, and SaltStack.

We use Ansible by defining a Playbook which defines a set of tasks to perform and a set of hosts to target. When we apply the Playbook, each task will be applied to each host. Some common use cases include:

• Executing remote commands
• Copying and downloading files
• Interaction with cloud providers
• User, group, and permission management

In order to create some measure of isolation and reuse, a developer can break a Playbook into one or many Roles. For example, if we primarily deploy Ruby applications, we might create a role called webapp_prerequisites which will prepare a host to run a Ruby application by installing the desired Ruby version and installing HAProxy to use as a local HTTP proxy. (For a brief discussion on why we would want to do this, see 2). This Role can then be referenced by each of the Playbooks that install our individual Ruby applications.

If you want to follow along with the examples below and setup your environment, see 7.

Creating a Role

Let’s create a simple role that installs Ruby, so that we can run a simple Sinatra or Rails application, and installs HAProxy to proxy requests to the application. In the Roles directory of our Ansible installation, we will create a directory webapp_prerequisites that contains a file tasks/main.yml with the following contents:

---
- name: Install Ruby
  package:
    name: ruby
    state: present
- name: Install HAProxy
  package:
    name: haproxy
    state: present

This YAML file contains a list of tasks that will install Ruby and HAProxy a system with some caveats 3. You can follow along with the code in this article via this repository 8.

HAProxy Virtual IP Failover

In order to demonstrate a more complex Role, we will create a Role that sets up two HAProxy hosts. One of the hosts will be active, and one will be a hot standby. The active HAProxy host will be bound to a virtual IP (VIP), and if there is some problem with the active host, the system with failover to the second HAProxy. In this scenario, the second host will claim the VIP.

Keepalived is a daemon that provides generic failover capabilities. We can configure it to monitor for the presence of an haproxy process. If one isn’t found, Keepalived peers can communicate with each other in order to coordinate assigning the VIP.

First, we have to install HAProxy on each host. We will make the assumption we’re running on CentOS. We also want the HAProxy service to be enabled and started.

- name: Install haproxy
  package:
    name: haproxy
- name: Enable haproxy
  systemd:
    name: haproxy
    state: started
    enabled: true

We typically want HAProxy to proxy another sevice. For these purposes, we’ll assume we’re proxying http://example.com/. We’ll create a configuration file to transfer to the remote host.

# haproxy.cfg

defaults
  mode http
  
frontend my_frontend
  bind *:80
  default_backend my_backend

backend my_backend
  # We need to send this request header otherwise
  # the upstream server won't respond.
  http-request set-header Host example.com
  server example example.com:80

We need a task to copy this configuration file to the appropriate location on the remote host:

- name: Copy haproxy.cfg
  copy:
    src: haproxy.cfg
    dest: /etc/haproxy/haproxy.cfg
  notify: reload haproxy

Whenever the configuration file is changed by our copy task, we want to queue a reload of the haproxy service. 6

Having configured HAProxy, we need to install and configure Keepalived.

- name: Install keepalived
  package:
    name: keepalived
- name: Setup keepalived service
  systemd:
    name: keepalived
    state: started
    enabled: true

Finally, we have to configure Keepalived with a Jinja template file. 1 It’s somewhat involved but the gist is we want to configure Keepalived to claim the VIP whenever HAProxy goes down. We templatize the configuration file because it will differ between host1 and host2. 9

We also need an Ansible task to create this file on the hosts:

- name: Create keepalived configuration
  template:
    src: keepalived.conf.j2
    dest: /etc/keepalived/keepalived.conf

That completes the role. For the full source see 8. Now that we’re done, how do we know it works? We can run it against a host we create manually but an even better way to do things is to create an automated test.

Testing Roles

Just like other pieces of software, Ansible Roles can be tested to confirm their operation. The reasons you might want an automated test are described in a previous blog post. It’s important to test Roles in some way because they often provide the base components of your system like runtimes, HTTP servers, and file and directory structures. Especially if multiple people are working on the same Role, the risk of adding the wrong configuration to a file or breaking a Jinja configuration template is high.

The standard way to test Ansible Roles is using Molecule. Molecule is a testing framework designed for Ansible. It works by adding a molecule directory to each Role, which contains instructions for how to provision testing hosts, Playbooks to apply the Role, and tests to run against each provisioned testing host. If you have an existing Role, this directory can be automatically generated by installing molecule, then running the following command inside the Role directory. For example, we can add molecule to the webapp_prerequisites Role by running the following command from inside the webapp_prerequisites directory.

% molecule init scenario \
  --driver-name vagrant \
  --scenario-name default \
    --role-name webapp_prerequisites

In the command above, we tell molecule that we want to create a new testing scenario called default, and to provision testing hosts using Vagrant. Vagrant makes it easy to programmatically provision virtual machines (VMs). 4

Afterwards, from inside the Role directory, we can run the tests using molecule test. The molecule test command will provision the testing infrastructure, in this case VMs running on VirtualBox, prepare and configure the hosts using Ansible, run tests against it using Python’s [testinfra](https://testinfra.readthedocs.io/en/latest/) framework, then destroy the testing infrastructure, reporting the results of the test. This can take quite awhile when using virtual machines, so when I am in the process of developing a Role, I use molecule verify, which will run all those steps without destroying the infrastructure. I can repeatedly run verify as I develop the Role.

Given the example Role definition above, we may want to test that HAProxy has been properly installed by running a simple command and checking its exit code. We can login to an instance provisioned by Molecule using molecule login.

We first run a command that should return the version of HAProxy that is installed, then we echo the exit code stored in $?.

% molecule login
--> Validating schema /.../ansible/roles/ecri.haproxy/molecule/default/molecule.yml.
Validation completed successfully.
[vagrant@instance ~]$ haproxy -v
HA-Proxy version 1.5.18 2016/05/10
Copyright 2000-2016 Willy Tarreau <willy@haproxy.org>
[vagrant@instance ~]$ echo $?
0

We can have Molecule test this automatically by writing a test using the testinfra Python provisioner testing framework. In this article, we’ll be using testinfra 1.14.1. In molecule/tests/test_default.py, we can add the following test, that does the same thing we did manually above.

def test_haproxy_installed(host):
    cmd = host.run("/usr/sbin/haproxy -v")
    assert cmd.rc == 0

After running molecule verify, we get a message like

tests/test_default.py::test_haproxy_installed[ansible://instance] PASSED [100%]

    =========================== 1 passed in 1.95 seconds ===========================
Verifier completed successfully.

testinfra has an API for retrieving hosts, running commands on those hosts, and obtaining information about the host (facts). For example, we can run a simple test to see if HAProxy is running and enabled on start with the following code:

def test_services_running_and_enabled(host):
    service = host.service('haproxy')
    assert service.is_running
    assert service.is_enabled

We can run arbitrary commands on the host, retrieving the stdout, stderr, and return code. This gives us a lot of flexibility in running our tests. For example, to determine if HAProxy is serving requests, we can use curl to test connectivity while logged onto the host:

def test_redirection(host):
    cmd = host.run("curl -v http://localhost")
    assert "HTTP/1.1 200" in cmd.stderr

In this test, we confirm that something is responding on the default http port 80, and that it responds with a 200 status code. If it’s present, the text will be in the stderr of the command. We use Python’s assert statement that will throw an error if the following condition is false.

Multi-machine testing

The test above is useful, and you can imagine the more complicated tests we can create to verify an instance. Still more complex scenarios arise in the case of multi-host testing.

In order to spin up multiple VMs, we modify the molecule.yml definition to include multiple platforms:

platforms:
  - name: host1
    box: bento/centos-7.5
    instance_raw_config_args: 
      - 'vm.network :private_network, ip: "192.168.3.9"'
  - name: host2
    box: bento/centos-7.5
    instance_raw_config_args: 
      - 'vm.network :private_network, ip: "192.168.3.10"'

We assign them static IPs 192.168.3.9 and 192.168.3.10 in a private network so that they can communicate with each other. A private network limits access to the VM to only the VM host machine 5.

When we run tests involving multiple hosts, we may need to setup the test by performing actions across hosts. testinfra will run the tests for each host, so we need a way to get a reference to a host that is not the one being tested. For this article, we will call the host currently being tested by testinfra the current testing host. testinfra typically passes a Host instance to each test method as the first argument:

def test_something(host): # <-- the host arg is a Host instance
  # test definition

We can use this Host instance to retrieve other hosts we previously defined in the platforms section of molecule.yml. We call host.get_host with a URI that refers to the host we want. When we are provisioning hosts with Ansible, it looks something like this:

my_host = host.get_host("ansible://my_host_name?ansible_inventory=" + 
  os.environ['MOLECULE_INVENTORY_FILE'])

get_host takes a URI whose scheme in the provisioner ansible. my_host_name is one of the host names defined in platforms. We tell Ansible the location of the inventory file that Molecule has generated via the ansible_inventory query parameter. Ansible requires a list of hosts to operate on, which is commonly referred to as the inventory.

ansible://my_host_name?ansbile_inventory=/tmp/inventory.txt
└─┬──┘   └─┬──────┘ └───────────────────────────┬┘
scheme      path                            query parameter

You may want to only run the test from a single host. Although I don’t know of a way to do this, we can skip tests when running on the wrong host. For example, if we only want a test to run against host1, then we can check the hostname of the current testing host, and return from the test method if it doesn’t match host1.

def test_only_host1(host):
  ansible_facts = host.ansible.get_variables()
  if ansible_facts['inventory_hostname'] != 'host1':
    return

In this case, we are using Ansible, so we request the facts about the host from Ansible. This includes the inventory_hostname, which is the hostname that Ansible refers to the host by. This may be something different than what the host uses to refer to itself.

In order to test the HAProxy VIP Playbook, we want to create a test that will confirm that the VIP still responds after the master host1 stops responding. It doesn’t matter which host is the current testing host as long as we confirm that when the keepalived-master host1’s, HAProxy server goes down, whichever host is associated to the VIP is still serving requests. In the test, we will assume we AREN’T running on host1.

def test_vip_active_after_one_node_goes_down(host):
    # if this is not host1, return

    try:
        service_haproxy(host, 'stop')
        wait_awhile()
        test_vip()
    finally:
        # After the test, make sure that haproxy is started again
        service_haproxy(host, 'start')
        wait_awhile()

The test_vip function just does a local curl and hits the VIP:

def test_vip():
    import subprocess
    assert 0 == subprocess.call([ "/usr/bin/curl", "-vs", 
        "--connect-timeout", "1", "http://192.168.3.2"])

For detail on what service_haproxy and wait_awhile, which are somewhat self-explantory, see 10.

Conclusion

In this article, we learned a little about what Ansbile is and what it can do. We learned about how to create Roles, and how to test them. We also learned how to write a complex multi-host testing scenario using Molecule and test_infra.

Endnotes

1 1: Jinja is the template language that Ansible uses. It uses curly braces to denote variables, and has looping and conditional constructs.

2 1: Local HTTP Proxies are useful as middleware to provide functionality that we might not want to build into our application. For example, HAProxy has features related to routing, http header injection, Lua-scripting, SSL, and throttling built into it can provide to an application without the need to make the application more complex by building it into the application. For example, if the application needed simple basic authentication, this could be provided by a local HAProxy instance proxying the application. This is made even more relevant when one uses containers. Using containers, it’s incredibly simple to add sidecar containers that add functionality.

3 1: The method we use to install Ruby and HAProxy results in versions that are dependent upon the underlying package manager and repositories installed on the host. For example, CentOS 7.5’s default repositories only include up to HAProxy 1.5, but the most recent version as of this writing is HAProxy 1.8.

4 1: Vagrant is useful for creating VMs for development. Vagrant itself is compatible with a number of local and remote VM providers such as VirtualBox, VMWare products, and Amazon Web Services (AWS).

5 1: The terminology is a bit confusing, but in this article, we call a host some logical computer running an OS. This is different than a VM host which is a machine that runs Virtual Machines.

6 1: In Ansible, Handlers are used to queue actions to happen at a later time. We know we want to reload HAProxy if the configuration changes, but we also might want to reload HAProxy if we have a change in an SSL certificate. We don’t want to reload HAProxy twice, so we just queue this reload to happen later.

7 1: Setting Up Ansible

The source code at 8 provides a requirements.txt file which lists the Python dependencies needed to follow along with the example. We can use virtualenv to install these dependencies into an isolated python environment. On MacOS, we can use homebrew to install Python 3 via

% brew install python

After making sure it is on the PATH, we can confirm our Python version using:

% python -V
Python 3.7.2

Then, we can install virtualenv globally using pip.

pip install virtualenv

We can create a virtualenv environment and activate it using something like:

virtualenv .python
source .python/bin/activate

Afterwards, we can install any requirements provided in the requirements.txt like:

pip install -r requirements.txt

8 123: GitHub repository for examples: https://github.com/jamiely/ansible_haproxy_vip

9 1: Keepalived Jinja Configuration template: https://github.com/jamiely/ansible_haproxy_vip/blob/master/templates/keepalived.conf.j2

10 1: VIP test using test_infra: https://github.com/jamiely/ansible_haproxy_vip/blob/master/molecule/default/tests/test_default.py