Automation and tools

As discussed previously, automation is the best practice to achieve rapid software delivery and solves the complexity to manage many microservices. However, automation tools are not an ordinary IT/infrastructure applications such as Active Directory, BIND (DNS), and Sendmail (MTA). In order to achieve automation, there is an engineer who should have both developer skill set to write a code, especially scripting language, and infrastructure operator skill set such as VM, network, and storage.

DevOps is a clipped compound of development and operations that can have an ability to make automation processes such as Continuous Integration, Infrastructure as code, and Continuous Delivery. DevOps uses some DevOps tools to make these automation processes.

Continuous Integration tool

One of the popular VCS tools is Git (https://git-scm.com). The developer uses Git to check-in and check-out the code all the time. There are some hosting Git service: GitHub (https://github.com) and Bitbucket (https://bitbucket.org). It allows you to create and save your Git repositories and collaborate with other users. The following screenshot is a sample pull request on GitHub:

The build server has a lot of variation. Jenkins (https://jenkins.io) is one of well-established applications, which is the same as TeamCity (https://www.jetbrains.com/teamcity/). In addition to build server, you also have hosted services, the Software as a Service (SaaS) such as Codeship (https://codeship.com) and Travis CI (https://travis-ci.org). SaaS has the strength to integrate with other SaaS tools.

Build server is capable of invoking an external command such as a unit test program; therefore, build server is a key tool within CI pipeline.

The following screenshot is a sample build using Codeship; it checks out the code from GitHub and invokes Maven to build (mvn compile) and unit testing (mvn test):

Continuous Delivery tool

There are a variety of configuration management tools such as Puppet (https://puppet.com), Chef (https://www.chef.io), and Ansible (https://www.ansible.com), which are the most popular in configuration management.

AWS OpsWorks (https://aws.amazon.com/opsworks/) provides a managed Chef platform. The following screenshot is a Chef recipe (configuration) of installation of Amazon CloudWatch Log agent using AWS OpsWorks. It automates to install CloudWatch Log agent when launching an EC2 instance:

AWS CloudFormation (https://aws.amazon.com/cloudformation/) helps to achieve infrastructure as code. It supports the automation for AWS operation, for example, to perform the following functions:

1. Creating a VPC.
2. Creating a subnet on VPC.
3. Creating an internet gateway on VPC.
4. Creating a routing table to associate a subnet to the internet gateway.
5. Creating a security group.
6. Creating a VM instance.
7. Associating a security group to a VM instance.

The configuration of CloudFormation is written by JSON as shown in the following screenshot:

It supports parameterize, so it is easy to create an additional environment with different parameters (for example, VPC and CIDR) using a JSON file with the same configuration. In addition, it supports the update operation. So, if there is a need to change a part of the infrastructure, there's no need to recreate. CloudFormation can identify a delta of configuration and perform only the necessary infrastructure operations on behalf of you.

AWS CodeDeploy (https://aws.amazon.com/codedeploy/) is also a useful automation tool. But focus on software deployment. It allows the user to define. The following are some actions onto the YAML file:

1. Where to download and install.
2. How to stop the application.
3. How to install the application.
4. After installation, how to start and configure an application.

The following screenshot is an example of AWS CodeDeploy configuration file appspec.yml:

Monitoring and logging tool

Once you start to manage some microservices using a cloud infrastructure, there are some monitoring tools that help you to manage your servers.

Amazon CloudWatch is the built-in monitoring tool on AWS. No agent installation is needed; it automatically gathers some metrics from AWS instances and visualizes for DevOps. It also supports to set an alert based on the criteria that you set. The following screenshot is an Amazon CloudWatch metrics for EC2 instance:

Amazon CloudWatch also supports to gather an application log. It requires installing an agent on EC2 instance; however, centralized log management is useful when you need to start managing multiple microservice instances.

ELK is a popular combination of stack that stands for Elasticsearch (https://www.elastic.co/products/elasticsearch), Logstash (https://www.elastic.co/products/logstash), and Kibana (https://www.elastic.co/products/kibana). Logstash helps to aggregate the application log and transform to JSON format and then send to Elasticsearch.

Elasticsearch is a distributed JSON database. Kibana can visualize the data, which is stored on Elasticsearch. The following example is a Kibana, which shows Nginx access log:

Grafana (https://grafana.com) is another popular visualization tool. It used to be connected with time series database such as Graphite (https://graphiteapp.org) or InfluxDB (https://www.influxdata.com). Time series database is designed to store the data, which is flat and de-normalized numeric data such as CPU usage and network traffic. Unlike RDBMS, time series database has some optimization to save the data space and faster query for numeric data history. Most of DevOps monitoring tools are using time series database in the backend.

The following example is a Grafana that shows Message Queue Server statistics:

Communication tool

Once you start to use several DevOps tools as we saw earlier, you need to go back and forth to visit several consoles to check whether CI and CD pipelines work properly or not. For example, consider the following points:

1. Merge the source code to GitHub.
2. Trigger the new build on Jenkins.
3. Trigger AWS CodeDeploy to deploy the new version of the application.

These events need to be tracked by time sequence, and if there are some troubles, DevOps needs to discuss it with the developer and QA to handle the cases. However, there are some over-communication needs, because DevOps needs to capture the event one by one and then explain, probably via e-mail. It is not efficient and in the meantime the issue is still going on.

There are some communication tools that help to integrate these DevOps tools and anyone can join to look at the event and comment to each other. Slack (https://slack.com) and HipChat (https://www.hipchat.com) are the most popular communication tools.

These tools support to integrate to SaaS services so that DevOps can see the event on the single chat room. The following screenshot is a Slack chat room that integrates with Jenkins:

Public cloud

CI CD and automation work can be achieved easily when used with cloud technology. Especially public cloud API helps DevOps to come up with many CI CD tools. Public cloud such as Amazon Web Services (https://aws.amazon.com) and Google Cloud Platform (https://cloud.google.com) provides some APIs to DevOps to control the cloud infrastructure. DevOps can be a relief from capacity and resource limitation, just pay as you go whenever the resource is needed.

Public cloud will keep growing the same way as software development cycle and architecture design; these are best friends and the important key to achieve your application/service to success.

The following screenshot is a web console for Amazon Web Services:

Google Cloud Platform also has a web console as shown here:

Both cloud services have a free trial period that DevOps engineer can use to try and understand the benefits of cloud infrastructure.