Taking Ansible apart

Unless you have been living under a rock, you have probably already heard about Ansible and how it can make your job of maintaining company infrastructure easier. But have you ever asked yourself what exactly Ansible is and how it works? We have and this is what we discovered.

Ansible components

When we install Ansible, we get two rather different sets of things:

1. an ever expanding collection of plugins that implement the actual Ansible functionality (modules, connection plugins, etc.) and
2. a few command line tools like ansible-playbook and ansible that herd aforementioned plugins.

It may feel a bit underwhelming at first, but as we will quickly learn, the way those pieces fit together is quite intricate.

Most of the time, when we think about Ansible, we should think about playbooks and ansible-playbook command. But not today. Today, we will break all the rules and use ansible command (you know, like real hackers steampunks ;).

Stripping Ansible down to its core

We will start by running this command:

$ ansible -m ping localhost

This command instructs Ansible to use the module to test if we have python available on our computer. And yes, we know doing this is a bit silly, since Ansible itself is written in python, but we can still learn a thing or two about Ansible while doing silly things.

If nothing went awry, executed command printed something similar to this:

 [WARNING]: Unable to parse /etc/ansible/hosts as an inventory source
 [WARNING]: No inventory was parsed, only implicit localhost is available
 [WARNING]: provided hosts list is empty, only localhost is available.
            Note that the implicit localhost does not match 'all'
localhost | SUCCESS => {
    "changed": false,
    "ping": "pong"

We will ignore the warnings for now, since we are steampunks and we know what we are doing ;) Also, those warnings will be gone by the time we get to the end of this post, so just keep calm and hack on.

This all seems quite straightforward and simple, until you start to question yourself: What exactly happened when we ran the previous command?

What using an Ansible module means

To see what actually happens when we run the ansible command, we will open a verbose valve on Ansible by adding a -vvv to the command line. This will allow Ansible to vent off some steam and tell us exactly what it is doing.

$ ansible -vvv -m ping localhost

Somewhere in the middle of the previous command’s output, we will find some lines, similar to this (we added a few newlines and shortened some file names in order to make this listing a bit neater):

<127.0.0.1> EXEC /bin/sh -c '(
     umask 77
  && mkdir -p "`echo /home/tadej/.ansible/tmp/ansible-tmp-154`"
  && echo ansible-tmp-154="`echo /home/tadej/.ansible/tmp/ansible-tmp-154`"
) && sleep 0'
Using module file /usr/lib/python/site-packages/ansible/modules/system/ping.py
<127.0.0.1> PUT /home/tadej/.ansible/tmp/ansible-local-186/tmpiqdxnirz
             TO /home/tadej/.ansible/tmp/ansible-tmp-154/AnsiballZ_ping.py
<127.0.0.1> EXEC /bin/sh -c '
     chmod u+x /home/tadej/.ansible/tmp/ansible-tmp-154/
               /home/tadej/.ansible/tmp/ansible-tmp-154/AnsiballZ_ping.py
  && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '
     /usr/bin/python /home/tadej/.ansible/tmp/ansible-tmp-154/AnsiballZ_ping.py
  && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '
     rm -f -r /home/tadej/.ansible/tmp/ansible-tmp-154/ > /dev/null 2>&1
  && sleep 0'

First EXEC line indicates that Ansible created a workspace on remote host for storing temporary files during the task execution. Next, Ansible transferred something that sounds like a less popular cousin of Dragon Ball to the remote host and executed it. And just like any other well-behaved program, Ansible cleaned after itself. Good boy Ansible ;)

But what is that ball and what process created it? What thing is generating all those shell commands? And what is responsible for copying those balls around? It seems that making Ansible more verbose only complicated the matters, so we better start explaining things ;)

Start the ball rolling

Whenever we use any Ansible module to bring our remote host into desired state, we are actually using a bundle of Ansible plugins that together make sure requested actions are executed on the remote host: a strategy plugin action plugin connection plugin shell plugin and a module (which is also Ansible plugin under the hood) for performing our task at hand.

Strategy plugin is responsible for scheduling the task execution on the host from the inventory. By default, ansible-playbook executes first task from the playbook on all hosts before starting the second one. But since we are only executing a single task today, strategy plugin has no real work to do.

Action plugin could actually be considered a part of the module that is executed on the control node (the computer running the ansible program). There are three main tasks that action plugin performs:

1. Substituting the variables in the module arguments.
2. Packaging the module sources into standalone ball of source code that will be executed on the remote host.
3. Coordinating the transfer and execution of the code on the remote host.

Connection plugin is responsible for transferring the files to and from the remote host and executing commands on the remote host. Probably the most well-known connection plugin is the ssh plugin that is used by default if we do not do anything funky in our playbooks and/or inventory files. But there are other connection plugins available that know how to connect to other kinds of remote hosts like for running tasks in OpenShift pods.

It should come as no surprise that shell plugin is responsible for composing the commands. These commands are then passed to the connection plugin that executes them on the remote host.

In our example Ansible run, the actual plugins that were used are:

1. linear strategy plugin that orchestrated the task execution,
2. normal action plugin that is used by default if there is no action plugin with the same name as the module being used,
3. local connection plugin because our remote host is localhost and
4. sh shell plugin because our remote host is a GNU/Linux machine.

Normal plugin first instructed the shell plugin to generate commands for creating temporary folders on the remote host and then used the local plugin to execute them. Next, normal plugin took the ping module source code with all of its dependencies and wrapped the whole thing into a standalone executable. That executable was then transferred to the remote host and executed by the local plugin.

Hopefully things make more sense now. All this being out of the way, we are ready now to tackle those warnings from the first command execution.

Making remote host remote

When we first run ansible command, we were kindly reminded that if we would like to run things on remote hosts, we would need to provide an inventory file. So let us create one that contains the following entries:

host_a ansible_host=10.10.43.151 ansible_user=centos
host_b ansible_host=10.10.43.152 ansible_user=ubuntu

We have declared two hosts here: a CentOS host at 10.10.43.151 and an Ubuntu host at 10.10.43.152 . Assuming that we stored this into file named inventory , we can now check for python interpreter on those two hosts by running

$ ansible -m ping -i inventory all
host_b | SUCCESS => {
    "changed": false,
    "ping": "pong"
host_a | SUCCESS => {
    "changed": false,
    "ping": "pong"

Steps that Ansible took while executing the above command are exactly the same as before. The only thing that changed is the connection plugin, since now we are connecting to remote hosts via .

We can verify this by again adding a -vvv to the command. Somewhere in the output of the comand, we should be able to find a (loooong) line similar to this next one:

<10.10.43.151> SSH: EXEC ssh -C -o ControlMaster=auto ... \
  '/bin/sh -c '"'"'/usr/bin/python /.../AnsiballZ_ping.py && sleep 0'"'"''

Bonus points for all of you who find the lines responsible for copying the data over to remote host.

Let us get the hell out of here

This has been one long post in which we briefly familiarized ourselves with the Ansible internals that are most commonly used. So we better stop before we make this even longer ;)

If this post tickled your fancy, you can reach us via Twitter Reddit . And do not forget to join us next time, when we will … well … do something … probably ;)

Cheers!