Ansible is a modern automation tool that makes our lives easier by helping us manage our servers, deployments, and infrastructure. We declare what we want and let Ansible do the hard work. Some of the things that Ansible can do are as follows:

• Install and configure software
• Manage users and databases
• Deploy applications
• Remote execution
• Manage Infrastructure as Code

We will focus on the Infrastructure as Code part of Ansible for a significant part of this book.

Ansible has certain distinct advantages over other similar tools.

• Ansible is agentless. So we do not need to install any software on the servers that are to be managed. It does require Python runtime on the servers and a SSH server on remote hosts.
• Ansible supports both push and pull modes. So we can execute Ansible code from a central control machine to make changes on remote machines or the remote machines can pull configuration from a well defined source periodically.
• Code for Ansible is written in YAML (http://yaml.org/), which stands for YAML Ain't Markup Language. Ansible did not create and manage a language (or DSL) from scratch. YAML is easy to read, write, and understand. This makes up most of Ansible code's self documentation and reduces the learning curve of Ansible significantly.
• Ansible does not try to re-invent the wheel. Hence it uses SSH as a transport and YAML as a Domain Specific Language (DSL). In typical cases, there are two entities involved, a system (A) where the playbook execution is initiated, and another system (B), usually remote, which is configured using Ansible:

In a nutshell, Ansible helps to manage various components of servers, deployments and infrastructure in a repeatable manner. Its self-documenting nature helps with understanding and auditing true nature of infrastructure.

Traditionally, infrastructure has been managed manually. At best, there would be a user interface which could assist in creating and configuring compute instances. For most of the users who begin their journey with a cloud, a web-based dashboard is the first and most convenient way to interact. However, such a manual method is error prone. Some of the most commonly faced problems are:

• Requirement for more personnel to manage infrastructure round the clock
• Probability of errors and inconsistencies due to human involvement
• Lack of repeatability and auditability

Creating Infrastructure as Code addresses these concerns and helps in more than one way. A well maintained code base will allow us to refer to the infrastructure state, not only at the present but also at various points in the past.

Ansible helps us code various aspects of infrastructure including provisioning, configuring, and eventually, retiring. Ansible supports coding over 20 cloud providers and self-managed infrastructure setups. Due to its open nature, existing providers can be enhanced and customized and new providers can be added easily.

Once we start managing Infrastructure as Code, we open ourselves to the possibility of a lot of automation. While this book focuses on creating and managing the infrastructure, the possibilities are limitless. We can:

• Raise an alarm if a critical machine becomes unreachable.
• Personnel who do not have access to infrastructure can still help by coding the infrastructure. A code review exercise could help to enforce best practices.
• We can scale infrastructure dynamically based on our requirements.
• In case of a disaster, we can create replacements quickly.
• Passing knowledge of best practices within and outside the organization becomes easier.

Throughout this book, we will create our infrastructure from Ansible code and demonstrate its usability and repeatability.

Before we start diving into the Ansible world, we need to know some basics:

• Inventory: We need to have a list of hosts that we want to manage. Inventory is that list. In its simplest form, this can be a text file created manually which just lists the IP addresses of the servers. This is usually enough for small infrastructure or if the infrastructure is static in nature. It follows ini syntax and a typical inventory would look like this:

[webservers]
server1
[application]
server1
server2

If our infrastructure is dynamic, where we add and remove servers frequently, we can use dynamic inventory. This would allow us to generate the inventory in real time. Ansible provides dynamic inventory scripts for many cloud providers and allows us to create dynamic inventory scripts as per our need for non-standard setups. We will use dynamic inventory in this book since it is better suited to cloud based environments.

• Modules: Ansible modules are executable plugins that get the real job done. Ansible comes with thousands of modules which can do tasks from installing packages to creating a server. Most of the modules accept parameters based upon which they take suitable actions to move the server towards a desired state. While this book uses modules primarily in the YAML code, it is possible to use modules in command line as an argument to Ansible ad hoc command.
• Tasks: A task is a call to an Ansible module along with all the requirements like parameters, variables etc. For example, a task may call upon the template module to read a Jinja template and set of variables, and generate the desired state of a file for the remote server.
• Roles: Ansible roles describe the particular role that a server is going to play. It consists of YAML code which defines tasks. It also consists of the dependencies for the execution of tasks like required files, templates and variables.
• Playbooks: A playbook is a YAML file where we associate roles and hosts. It is possible to write the tasks in the playbook itself and not use roles but we would strongly discourage this practice. For the sake of readability and better code management, we suggest that playbooks should be kept small with just the name of the hosts or groups to target as defined in inventory and calls to the roles.
• Variables: Just like any other programming language, Ansible also has good old-fashioned variables. They hold values which can be used in tasks and templates to perform certain actions. Variables can be created before execution in a file or generated and modified during runtime. A very common use case is to invoke certain tasks if a variable holds a certain value or to store the output of a task in a variable and use it in one of the subsequent tasks.

There are many ways to install Ansible. Most of the Linux distributions has Ansible packages in their repositories. Compiling from source is also an option. However, for the sake of uniformity, we are going to use Python pip to install the same version of Ansible for all the readers.

The following command will fetch the Ansible source and install Ansible 2.4.0.0 on our working machine. We have used this version of Ansible throughout the book and we urge our readers to install the same:

$ sudo pip install ansible==2.4.0.0

The simplest form in which we can use Ansible with Ansible ad hoc command line tool. We can execute the tasks using modules without actually writing code in the file. This is great for quick testing or for one off tasks but we should not turn this into a habit since this type of usage is not easily documented and auditable.

We just have to use the Ansible command and pass the ping module as an argument to parameter -m. A successful execution will return the string pong. It signifies that Ansible can reach the server and execute tasks, subject to the authorization level of the user, of course:

$ ansible localhost -m ping
localhost | SUCCESS => {
    "changed": false,
    "failed": false,
    "ping": "pong"
}

Under normal circumstances, users execute the ansible-playbook command from a system, say A. This system has inventory, playbooks, roles, variable definitions and other information required to configure a remote system, say B, to a desired state.

When we talk about building infrastructure using Ansible, things change a bit. Now, we are not configuring a remote system. We are actually interacting with a cloud provider to create or allocate certain resources to us. We may, at a later point in time, choose to configure these resources using Ansible as well. Interacting with a cloud provider is slightly different from executing a regular playbook. There are two important points that we need to keep in mind:

• A lot of the tasks will execute on the local machine and will interact with API provided by a cloud provider. In principle, we won't need SSH setup because, in typical cases, requests will go from our local machine to the cloud provider using HTTPS.
• The cloud provider will need to authenticate and authorize our requests. Usually this is done by providing a set of secrets, or keys, or tokens. Since these tokens are sensitive, we should learn a little bit about Ansible Vault.

Secret management is an important aspect of any configuration management tool. Ansible comes with a tool called Ansible Vault which encrypts secrets (technically, it can encrypt any arbitrary file but we will focus on secrets) at rest with 256 bit AES encryption. These secrets can be used in tasks in various ways.

To understand this better, let us create a sample secret and use it in a task.

We will begin with a standard variable file, let us call it secret.yml, in Ansible:

---
mysecret: secret-value

To use this in a playbook, we can include the file as a variable and call it in a task:

---
- hosts: localhost
  tasks:
    - name: include secret
      include_vars: secret.yml

    - name: get value
      debug:
        msg: "The value is: {{ mysecret }}"

Let us run our playbook to verify that everything is good:

$ ansible-playbook playbook.yml
PLAY [localhost] ***************************************************************
TASK [setup] *******************************************************************
ok: [localhost]
TASK [include secret] **********************************************************
ok: [localhost]
TASK [get value] ***************************************************************
ok: [localhost] => {
"msg": "The value is: secret-value"
}
PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0

Our goal is to protect the content of secret.yml. So let us use ansible-vault and encrypt it:

$ ansible-vault encrypt secret.yml
Vault password:
Encryption successful

Now the content of secret.yml should look something like this:

$ANSIBLE_VAULT;1.1;AES256
64656138356263336432653663323966373961363637383035393631383963643363343162393764
6634663662333863373937373139326230326366643862390a643435663237333832366336323861
31666565333937343333373133353838396166356233316435643363356161366536356230396534
3038316565336630630a393938613764616530336565653866346130666466346130633563346564
33313230336265383532313033653237643662616437636263633039373065346537

Executing the playbook like before will fail because our variable file is encrypted. Ansible provides a way to read encrypted files on the fly without decrypting it on the disk. The flag, --ask-vault-pass, will request the password from and execute the playbook normally when provided with the correct password:

$ ansible-playbook --ask-vault-pass playbook.yml
Vault password:
PLAY [localhost] ***************************************************************
TASK [setup] *******************************************************************
ok: [localhost]
TASK [include secret] **********************************************************
ok: [localhost]
TASK [get value] ***************************************************************
ok: [localhost] => {
"msg": "The value is: secret-value"
}
PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0

We will be using Ansible Vault throughout this book to store secrets.

Throughout the book, we will give examples of how to deploy an application on the infrastructure created by the Ansible playbooks. We have written a simple phone book application using Python's flask framework (http://flask.pocoo.org). The phone book application listens on port 8080 and we can use any browser to use the phone book. The app has two variations, one uses SQLite as a database whereas the other one uses MySQL. The application code remains the same, we have just used different databases to demonstrate the application running in a single compute instance and it running across multiple instances or even different components of a cloud provider.

The application code can be obtained from:

• Phone book SQLite: https://github.com/ansible-cookbook/phonebook-sqlite
• Phone book MySQL: https://github.com/ansible-cookbook/phonebook-mysql

The application deployment can be done using Ansible. If we are going to deploy the application using SQLite then the following tasks for the phonebook role are good enough:

---
- name: install epel repository
  package:
    name: epel-release
    state: present

- name: install dependencies
  package:
    name: "{{ item }}"
    state: present
  with_items:
  - git
  - python-pip
  - gcc
  - python-devel

- name: install python libraries
  pip:
    name: "{{ item }}"
    state: present
  with_items:
  - flask
  - flask-sqlalchemy
  - flask-migrate
  - uwsgi

- name: get the application code
  git:
    repo: git@github.com:ansible-cookbook/phonebook-sqlite.git
    dest: /opt/phone-book

- name: upload systemd unit file
  copy:
    src: phone-book.service
    dest: /etc/systemd/system/phone-book.service

- name: start phonebook
  systemd:
    state: started
    daemon_reload: yes
    name: phone-book
    enabled: yes

In the case of MySQL, we need to add some more tasks and information to work with Ansible:

---
- name: include secrets
  include_vars: secrets.yml

- name: install epel repository
  package:
    name: epel-release
    state: present

- name: install dependencies
  package:
    name: "{{ item }}"
    state: present
    with_items:
    - git
    - python-pip
    - gcc
    - python-devel
    - mysql-devel 

- name: install python libraries
  pip:
    name: "{{ item }}"
    state: present
  with_items:
  - flask
  - flask-sqlalchemy
  - flask-migrate
  - uwsgi
  - MySQL-python

- name: get the application code
  git:
    repo: git@github.com:ansible-cookbook/phonebook-mysql.git
    dest: /opt/phone-book
    force: yes

- name: upload systemd unit file
  copy:
    src: phone-book.service
    dest: /etc/systemd/system/phone-book.service

- name: upload app config file
  template:
    src: config.py
    dest: /opt/phone-book/config.py

- name: create phonebook database
  mysql_db:
    name: phonebook     
    state: present
    login_host: "{{ mysql_host }}"
    login_user: root
    login_password: "{{ mysql_root_password }}"

- name: create app user for phonebook database
  mysql_user:
    name: app
    password: "{{ mysql_app_password }}"
    priv: 'phonebook.*:ALL'
    host: "%" 
    state: present
    login_host: "{{ mysql_host }}"
    login_user: root
    login_password: "{{ mysql_root_password }}"

- name: start phonebook
  systemd:
    state: started
    daemon_reload: yes
    name: phone-book
    enabled: yes

Accordingly, we will create a secrets.yml in vars directory and encrypt it using ansible-vault. The unencrypted data will look like this:

---
mysql_app_password: appSecretPassword
mysql_root_password: secretPassword
mysql_host: 35.199.168.191

The phone-book.service will take care of initializing the database and running the uwsgi server for serving the application for both SQLite and MySQL based setups:

[Unit]
Description=Simple Phone Book

[Service]
WorkingDirectory=/opt/phone-book
ExecStartPre=/bin/bash /opt/phone-book/init.sh
ExecStart=/usr/bin/uwsgi --http-socket 0.0.0.0:8080 --manage-script-name --mount /phonebook=app:app
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Throughout the coming chapters, we will use this role to deploy our phone book application.

We have talked about dynamic inventory a little bit in this chapter. Throughout this book, in every chapter, we are going to talk about and use dynamic inventory. So let us explore the concept in a bit more depth.

Reiterating what we wrote earlier, dynamic inventory is useful for infrastructures that are dynamic in nature or for cases where we do not want to or cannot maintain a static inventory. Dynamic inventory queries a datasource and builds the inventory in real time. For the sake of this book, we will query cloud providers to get data and build the inventory. Ansible provides dynamic inventory scripts for most of the popular cloud providers.

However, it is simple to create a dynamic inventory script by ourselves. Any executable script that can return a JSON with a list of inventory host groups and hosts in a predetermined format, when passed with a parameter --list can be used as an inventory script. A very simple inventory would output something like this:

{
     "application": ["10.0.0.11", "10.0.0.12"],
      "database": ["10.0.1.11"]
}

More elaborate inventory scripts would output much more information like instance tags, names, operating systems, geographical locations, and, also known as host facts.

To present a realistic example, we have created a simple inventory script for Amazon Web Service in Python. The code is available on GitHub (https://github.com/ansible-cookbook/ec2_tags_inventory):

#!/usr/bin/env python
import boto3
import json
import ConfigParser
import os

def get_address(instance):
    if "PublicIpAddress" in instance:
          address = instance["PublicIpAddress"]
    else:
          address = instance["PrivateIpAddress"]
    return address

if os.path.isfile('ec2.ini'):
   config_path = 'ec2.ini'
elif os.path.isfile(os.path.expanduser('~/ec2.ini')):
   config_path = os.path.expanduser('~/ec2.ini')

config = ConfigParser.ConfigParser()
config.read(config_path)
id = config.get("credentials", "aws_access_key_id", raw=True)
key = config.get("credentials", "aws_secret_access_key", raw=True)

client = boto3.client('ec2', aws_access_key_id = id, aws_secret_access_key = key, region_name="us-east-1")

inventory = {}

reservations = client.describe_instances()['Reservations']
for reservation in reservations:
   for instance in reservation['Instances']:
         address = get_address(instance)
         for tag in instance['Tags']:
               if tag['Key'] == "ansible_role":
                     roles = tag['Value'].split(",")
                     for role in roles:
                           if role in inventory:
                                 inventory[role].append(address)
                           else:
                                 inventory[role] = [address]
print json.dumps(inventory)

This script reads a file called ec2.ini for AWS access and secret key. For the sake of simplicity, we have hardcoded the region to us-east-1 but this can be changed easily. The script goes through AWS EC2 in the us-east-1 region and looks for any instance that has a tag with the name ansible_role and any valid value like webserver or database. It will add the IP addresses of those instances to the Python dictionary variable called inventory. In the end, this variable is dumped as JSON as output.

We can test this by executing:

$ python ec2_tags_inventory.py --list
{"application": ["10.0.0.11", "10.0.0.12"], "database": ["10.0.1.11"]}

Note that output may vary depending on the instances that are tagged in EC2. To use this in an Ansible command, we need to make it executable and just pass the script instead of inventory file to -i flag like this:

$ chmod +x ec2_tags_inventory.py
$ ansible -i ec2_tags_inventory.py database -m ping
database | SUCCESS => {
"changed": false,
"failed": false,
"ping": "pong"
}

Needless to say, this is a very simple example and the actual dynamic inventory script provided by Ansible is much more comprehensive and it looks beyond EC2 to other services, such as RDS.