Ansible Idempotency and Configuration Drift

Idempotency at all times when building Ansible tasks

Overview

In general,  idempotence is “the property of certain operations in mathematics and computer science that can be applied multiple times without changing the result beyond the initial application”.  For Ansible it means after 1 run of a playbook to set things to a desired state, further runs of the same playbook should result in 0 changes.  In simplest terms, idempotency means you can be sure of a consistent state in your environment.

Configuration Drift is where servers in an infrastructure become more and more different as time goes on.  So the state of the machine deviates, or drifts, from the baseline due to manual changes and updates.

To help combat this configuration drift, two approaches can be taken.  1) Use an automated tool, such as Ansible, and run them frequently and repeatedly to keep machines in line 2) Rebuild machine instances frequently so they don’t have much time to drift from the baseline.

Removing or Minimizing Drift with Ansible

There are multiple ways to combat Configuration Drift with Ansible, such as using Ansible Tower to check frequently with playbooks that detect drift, then notify the appropriate resource through chat, email, etc and/or make an appropriate modification to keep it in line with the baseline.

Example 1

Ansible Modules are idempotent as we mentioned above. Modules will enforce a change in state only if there is a configuration drift.  Let’s say we have a nginx role that contains a default.conf.  We want to make sure the default configuration, which is in roles/nginx/files, is used on the instance.  If we launch an Ansible run after making a manual change then, while executing, it will compare the files inside our role with the one on the system, detect the configuration drift, and copy it over the changed file.  A handler could then be used to restart the service.  This keeps the system in a known state.

Example 2

Another example would be using “-check” and “-diff”.  Through Ansible Tower a scheduled playbook could run on each host or group in an infrastructure.  A report could then be emailed to a DevOps group showing all the playbooks that showed something would have changed.  In the ideal state the report would be empty.

Example 3

It is a good idea to test idempotency and make sure the playbooks remain idempotent.  I like this simple example below describing a non-idempotent and idempotent Bash:

Non-Idempotent Bash

echo “127.0.0.1 localhost” >> /etc/hosts

This would of course append the entry to /etc/hosts over and over again.

Idempotent Bash

If (!grep -q 127.0.0.1 /etc/hosts); then echo “127.0.0.1 localhost” >>  /etc/hosts; fi

Although it doesn’t check for commented lines, it does check for a portion of the line.  So let’s expand this to playbooks.

Ansible’s template or copy modules allow you to prepare an entire file upfront which gives you complete control over the file’s contents.  So when possible prefer the copy and template module over manipulating with the lineinfile module.  We can use lineinfile module to ensure idempotency at all times so that an Ansible task which introduces a change to your system must not report the same change when run a second time.   See the references below which describe cases for lineinefile more in depth.

So we are going to make the SSH daemon listen on IP Address 1.2.3.4.

First Iteration

– name: Listen on 1.2.3.4

  lineinfile: dest=/etc/ssh/sshd_config

              line=”ListenAddress 1.2.3.4″

              state=present

The line ListenAddress 1.2.3.4 will be added at the end of the sshd_config file because we did not specify where to put it.  While this achieves the change the approach is not good in that it is listening on the address 1.2.3.4 but may also be listening on other addresses we don’t know about.

Second Iteration

– name: Listen on 1.2.3.4

  lineinfile: dest=/etc/ssh/sshd_config

              line=”ListenAddress 1.2.3.4″

              insertafter=”^#?AddressFamily”

 

This task will add the line ListenAddress 1.2.3.4 directly under the line starting with AddressFamily (comment or not).  The goal was to make the SSH daemon listen on only the IP address we specify.  So 2 task were done:

  • Remove all occurrences of ListenAddress that do not read ListenAddrtess 1.2.3.4
  • Then we have to add the line ListenAddress 1.2.3.4 at a specific place

 

Deeper Dive Resources

http://tylerturk.com/testing-ansible-idempotency/

https://relativkreativ.at/articles/how-to-use-ansibles-lineinfile-module-in-a-bulletproof-way

https://www.scrye.com/wordpress/nirik/2015/01/25/ansible-idempotence-tips/

https://books.google.com/books