CM systems and CaC

Setting up and maintaining a system other than a hobbyist server (and even those, maybe, too) poses a serious challenge: how do you ensure that the system is installed and configured correctly and according to expectations? When you have to install a new server that is identical in configuration, how do you ensure that? In the past, a way of doing it was documenting the current configuration after the installation process was done. This would be a document describing the hardware, operating system, installed software, created users, and configuration applied. Any person who wanted to recreate it would have to follow steps to achieve the configuration described in the document.

The very next logical step is to write shell scripts that achieve the same goal with one additional improvement over the manual process: the scripts—properly written, tested, and maintained—do not require manual work, except, maybe, the initial system installation. But a properly set up environment would take care even of this.

The scripts, however, also have some defects or deficiencies. One of them is the fact that you need to account in your scripts for unfinished execution. This could happen for various reasons and would leave the system in a partially configured state. Executing the script again would perform all configuration actions from the start, sometimes leading to unexpected results. One way to account for incomplete runs would be to wrap every configuration action in a check, to see whether it had been performed previously. That would lead to the configuration script becoming larger and, eventually, evolving into a library of configuration functions and check functions.

The task of developing and maintaining such a tool can be daunting and will probably require a whole team. Still, the results are probably worth the effort.

Writing and maintaining documentation that describes the desired state of the system may, at first glance, be simpler and more desirable than the previously mentioned method of automating. The script cannot recover from an incomplete execution. The best it can do is inform the sysop about failure, log the error, and stop gracefully. Manually performed configuration allows the sysop to work around any obstacles and inadequacies in the procedure and edit the document to reflect the current state on the go.

Still, a properly developed and tested script turns out to be better. Let us enumerate the reasons:

• If the script executes without an error, it is guaranteed to perform actions without a mistake. Time and again, it has been proven that a human is the element most prone to errors in IT.
• If the script exits prematurely, the action of updating it to account for the new requirements is a perfect equivalent to updating the documentation.
• People are known to be pretty bad at maintaining documentation. The Holy Grail of programming is self-documenting code, rendering comments unnecessary, thus eliminating the risk of comments being out of sync with the code.
• The script can be executed on multiple systems at once, scaling very well, if not infinitely. Humans can perform the configuration of one system at a time with minimal risk of making a mistake.
• Configuration kept in the form of a script or program benefits from typical programming techniques, such as automated testing, dry runs, and static analysis. More so, keeping the code in a repository allows us to easily track a history of changes and integrate it with ticket-tracking tools.
• Code is unequivocal, which cannot be said about written language. A document may leave space for interpretation; a script won’t.
• Automating configuration lets you move to other, more interesting tasks, leaving the computers to do what they do best—performing repetitive, boring tasks well.

The world of programming and system administration has a tendency to turn small projects into larger ones with a vibrant community of developers and users. It was only a matter of time before CM systems were born. They take the burden of developing and managing portions of the code responsible for configuration actions off your shoulders. The CM system developers write the code, test it, and deem it stable. What you are left with is an action of writing configuration files or directives that tell the system what to do. Most of these systems will be able to cover the most popular platforms, allowing you to describe configuration once and run it with the same expected results on commercial Unix systems, such as AIX or Solaris, as on Linux or Windows.

Configuration files for these systems are easily stored in a version control system such as Git. They are easily understandable by a human, which allows for simple review by your colleagues. They can be checked for syntax errors by automated tools and allow you to concentrate on the most important part of the whole endeavor: the configuration.

This approach of keeping your configuration as a set of scripts or other data instead of a procedure to be followed manually is known as CaC.

The CaC approach is becoming more important as the number of systems to be managed grows and the demand for fast and efficient configuration scales up. In the world of DevOps, it is usual practice to set up tens and hundreds of systems a day: systems for developers, testers, and production systems to manage new levels of demand for the service. Managing it manually would be an impossible task. Well-implemented CaC allows to run this task with a click of a button. Thus, developers and testers can deploy their own systems without bothering sysops. Your task will be to develop, maintain, and test the configuration data.

If there is one thing sure in the world of programming, it is that there’s never going to be only one solution. The same goes for CM tools.

Alternatives for Ansible include SaltStack, Chef, Puppet, and CFEngine, which is the oldest one; its initial release date was 1993, so it’s 30 years old as of the time of writing this book. In general, those solutions differentiate between each other with a method of enforcing configuration (pull or push) and an approach of describing the system’s state (imperative or declarative).

Imperative means that we describe the state of the server with commands for the tool to perform. Imperative programming focuses on describing how a given tool operates step by step.

Declarative, on the other hand, means we focus on what the CaC tool should accomplish without specifying all the details of how it should achieve the result.

SaltStack

SaltStack is an open source CM tool that allows for the management of complex IT infrastructure at scale. It enables the automation of routine tasks such as package installation, user management, and software configuration, and is designed to work across a wide range of operating systems and platforms. SaltStack was founded by Thomas Hatch in 2011. The first release of SaltStack, version 0.8.0, was also made in 2011.

SaltStack works by utilizing a master-slave architecture, where a central salt-master communicates with salt-minions running on remote machines to execute commands and manage configurations. It operates in a pull method of enforcing configuration: minions pull the latest manifest from the master server.

Once the minion is installed and configured, we can use SaltStack to manage the server’s configuration. Here’s an example nginx.sls file that would install and configure nginx:

nginx:
  pkg.installed
/etc/nginx/sites-available/yourdomain.tld.conf:
  file.managed:
    - source: salt://nginx/yourdomain.tld.conf
    - user: root
    - group: root
    - mode: 644

In this example, the first line specifies that the nginx package should be installed on the target server. The next two lines define the configuration file for a hypothetical website, example.com, which is copied to /etc/nginx/sites-available/yourdomain.tld.conf.

To apply this state file to a server, we would use the state.apply command in the SaltStack command-line interface, specifying the name of the state file as the argument:

admin@myhome:~$ salt 'webserver' state.apply nginx

This would send the instructions in the nginx.sls file to the salt-minion running on the web server machine, which would execute the necessary steps to ensure that nginx is installed and configured correctly.

Chef

Chef is a powerful open source CM tool that allows users to automate the deployment and management of infrastructure, applications, and services. It was first released in 2009 by Opscode, which was later acquired by Chef Software Inc. Since then, Chef has been widely adopted by IT professionals and DevOps teams to streamline their workflows and reduce the time and effort required for managing complex systems.

Chef works by defining the desired state of an infrastructure in a set of code files, called cookbooks. A cookbook is a collection of recipes that describe how to install, configure, and manage a specific piece of software or service. Each recipe contains a series of resources, which are pre-built modules that can perform specific tasks, such as installing a package or configuring a file. Chef uses a declarative approach to CM, meaning that users define what they want the system to look like, and Chef takes care of the details of how to get there.

To install nginx using Chef, you would first need to create a cookbook that includes a recipe for installing nginx. This recipe would use the package resource to install the nginx package and the service resource to ensure that the nginx service is running. You could also use other resources, such as file, directory, or template, to configure nginx’s settings, depending on your requirements.

Once you had created the cookbook, you would upload it to a Chef server, which acts as a central repository for cookbooks and their associated metadata. You would then use Chef’s command-line tool, called knife, to configure the target system to use the cookbook. This involves associating the system with a Chef environment, which defines the set of cookbooks and their versions that should be applied to the system. You would then use the chef-client command to run the Chef client on the target system, which will download and apply the necessary cookbooks and recipes to bring the system into the desired state.

Here’s an example of installing and configuring nginx:

# Install Nginx package
package 'nginx'
# Configure Nginx service
service 'nginx' do
  action [:enable, :start]
end
# Configure Nginx site
template '/etc/nginx/sites-available/yourdomain.tld.conf' do
  source 'nginx-site.erb'
  owner 'root'
  group 'root'
  mode '0644'
  notifies :restart, 'service[nginx]'
end

This recipe uses three resources, as follows:

• package: This installs the nginx package using the default package manager on the system.
• service: This starts and enables the nginx service so that it will automatically start on boot and stay running.
• template: This creates a configuration file for nginx by generating it from a template file. The template file (nginx-site.erb) is written in Embedded Ruby (ERB) format and is located in the templates directory of the cookbook. The notifies attribute tells Chef to restart the nginx service if the configuration file changes.

Once you have created this recipe in a cookbook, you can use the knife command to upload the cookbook to a Chef server. You can then use the chef-client command to apply the recipe to a target system, which will install and configure nginx according to the recipe.

Puppet

Puppet is an open source CM tool that allows system administrators to automate the deployment, configuration, and management of infrastructure. It was created by Luke Kanies in 2005 and released under the Apache License 2.0.

Puppet works by defining the desired state of infrastructure resources in a declarative language, known as the Puppet language. Administrators can define the configuration of servers, applications, and other infrastructure components in Puppet code, which can then be applied consistently across multiple systems.

Puppet consists of a master server and multiple agent nodes. The master server acts as a central repository for Puppet code and configuration data, while the agent nodes execute the Puppet code and apply the desired state to the system.

Puppet has a robust ecosystem of modules, which are pre-written Puppet code that can be used to configure common infrastructure resources. These modules are available in Puppet Forge, a public repository of Puppet code.

Here’s an example Puppet manifest that installs nginx and creates a configuration file similar to what we did with SaltStack and Chef:

# Install Nginx
package { 'nginx':
  ensure => installed,
}
# Define the configuration template for the domain
file { '/etc/nginx/sites-available/yourdomain.tld.conf':
  content => template('nginx/yourdomain.tld.conf.erb'),
  owner   => 'root',
  group   => 'root',
  mode    => '0644',
  notify  => Service['nginx'],
}
# Enable the site by creating a symbolic link from sites-available to sites-enabled
file { '/etc/nginx/sites-enabled/yourdomain.tld.conf':
  ensure  => 'link',
  target  => '/etc/nginx/sites-available/yourdomain.tld.conf',
  require => File['/etc/nginx/sites-available/yourdomain.tld.conf'],
}
# Restart Nginx when the configuration changes
service { 'nginx':
  ensure     => running,
  enable     => true,
  subscribe  => File['/etc/nginx/sites-enabled/yourdomain.tld.conf'],
}

Once you’ve created a manifest and put it on the Puppet server, it will be picked up by the Puppet agent installed on your server and executed. Communication, the same as in SaltStack, is being secured by the TLS protocol using the same mechanism as the HTTPS servers on the internet.

The agent nodes run a Puppet agent process, which connects to the master server over TCP port 8140. The agent sends a certificate signing request (CSR) to the server, which the administrator must approve. Once the CSR is approved, the agent is granted access to the server’s Puppet configuration.

When the agent runs, it sends a request to the master server for its configuration. The server responds with a catalog of resources that should be applied to the node. The catalog is generated based on the Puppet code and manifests stored on the server, as well as any external data sources or hierarchies that are configured.

The agent then applies the catalog to the node, which involves making any necessary changes to the node’s configuration to ensure it matches the desired state defined in the catalog. This may involve installing packages, updating configuration files, or starting or stopping services.

The agent sends reports back to the server after applying the catalog, which can be used for monitoring and auditing purposes. The server can also use this information to detect changes to the node’s configuration that were not made through Puppet and to take corrective action if necessary.

CFEngine

CFEngine is an open source CM system that allows users to automate the deployment, configuration, and maintenance of IT systems. It was founded by Mark Burgess in 1993 and has since become a popular tool for managing large-scale IT infrastructures. CFEngine is known for its powerful and flexible language for describing system configurations and enforcing policies, making it a great choice for complex IT environments.

CFEngine’s first release was in 1994, making it one of the oldest CM tools in existence. Since then, CFEngine has undergone numerous updates and improvements to keep up with changing IT environments and emerging technologies. The latest release of CFEngine, version 3.18, includes features such as improved encryption, enhanced monitoring capabilities, and better support for cloud infrastructure.

CFEngine has gained popularity over the years due to its robust functionality, ease of use, and strong community support. It’s still being used today by many organizations and is actively developed, so it is a safe option to manage the configuration of your servers using this tool.

An example CFengine configuration will be presented here. It is, out of necessity, only a snipped and not a complete configuration:

##############################################################
# cf.main - for master infrastructure server
##################################################################
###
# BEGIN cf.main
###
control:
   access    = ( root )        # Only root should run this
   site      = ( main )
   domain    = ( example.com )
   sysadm    = ( [e-mail] )
   repository = ( /var/spool/cfengine )
   netmask   = ( 255.255.255.0 )
   timezone  = ( CET )
#################################################################
files:
  Prepare::
      /etc/motd              m=0644 r=0 o=root act=touch

In this section, we have explained what CaC is and why it is an important tool in the toolbelt of system administrators. We have briefly described the most popular tools available to you. In the next section, we will introduce our tool of choice—Ansible.