Mastering Chef

Chapter 1. Introduction to the Chef Ecosystem

Chef is a configuration management system written partly in Ruby and Erlang.

Before we begin our exciting journey towards becoming Chef masters, I think it would be prudent on our part to understand the underlying ecosystem.

The Chef ecosystem is primarily comprised of the following components:

WebUI: This is a Rails application that is used to view information about the Chef server over the Web.
ErChef: Prior to version 11.x, the Chef server API core (the code responsible for catering to requests by Knife or chef-client) was written in Ruby. However, since 11.x, this code has been rewritten in Erlang.
Bookshelf: This is used to store cookbooks content such as files, templates, and so on, that have been uploaded to chef-server as part of a cookbook version.
chef-solr: This is a wrapper around Apache Solr and is used to handle the REST API for indexing and search.
Rabbit MQ: This is used as a message queue for the Chef server. All items that are to be added to a search index repository are first added to a queue.
chef-expander: This is a piece of code that pulls messages from the RabbitMQ queue, processes them into a desired format, and finally posts them to Solr for indexing.
PostgreSQL: This is another major change since version 11.x. Earlier, CouchDB used to be the data storage; however, since version 11.x, PostgreSQL has become the data storage solution used by Chef.
chef-client: This is a Ruby application that runs on every machine that needs to be provisioned. It handles the task of authenticating with chef-server, registering nodes, synchronizing cookbooks, compiling resource collections, handling exceptions and notifications, and so on.
Knife: This is a Ruby application that provides an interface between a local chef repository and the Chef server.

The typical architecture of the Chef ecosystem can be understood by looking at the following figure:

Other than these components, we've chef-shell (shef), Ohai, and chef-solo that form an integral part of the chef ecosystem. We also have chef-zero, which is being adopted by people to quickly test their code or deploy chef code locally. It's a small, fast, and in-memory implementation of the Chef server and it helps developers to write a clean code without all the hooks that were earlier placed into the code to ensure that chef-solo can execute the code.

With the understanding of the Chef ecosystem, we will be covering the following topics in this chapter:

Different modes of running Chef
Terminology used in the world of Chef
The anatomy of a Chef run
Using the Chef Solo provisioner
Setting up a work environment

Different modes of running Chef

Chef can be executed under different modes. It's generally set up in a client-server fashion. However, if you were to just bootstrap your machine using Chef code, you don't need to worry about setting up a Chef server. Chef also provides a way of running as a standalone executable. If you are a developer writing a new piece of infrastructure code and want to test it, you can even run it in an IRB-like shell.

The most used mode of running Chef is the client-server model. In this model, we've a Chef server and an agent called chef-client that runs on machine(s) that need to be set up. The Chef client communicates with a chef-server and bootstraps the machine appropriately depending upon certain parameters, which we'll learn about shortly.

In a client-server architecture, the Chef ecosystem is comprised of a chef-server, which in turn is a name given to a set of services running on an instance (chef-server-web-ui, chef-solr, chef-expander, chef-core-api, and so on) and chef-client, which is an agent running on machines.

The chef-solo is the tool to be used if you just want to provision an instance using Chef. With chef-solo, we can do everything except for using the search functionality or accessing remote data bags that the chef-server provides. The chef-solo tool is expected to be deprecated in the near future and chef-zero is the expected way to run the code locally.

Shef is more like a debugging tool that allows you to set breakpoints within a recipe. It runs as an IRB session. It provides support for interactive debugging too.

By default, chef-shell loads in a standalone mode. However, it can also run as a chef-client and be used to verify the functionality of a Chef run. Set up chef-shell.rb with the same settings as those in knife.rb and run with the –z option:

$ chef-shell –z

We'll cover more about using Shef for debugging purposes later in this book.

The Chef server can either be set up privately, or you can choose a managed hosting service provided by Opscode. Here again, you've a choice of using an open source Chef or Enterprise Chef.

Enterprise Chef adds the following additional features on top of an open source Chef:

Enhanced management console
Centralized monitoring and reporting
Role-based access control
Push client runs

Terminology used in the world of Chef

Before jumping into a new territory, it's always wise to learn about the terminology used by the people already living in the environment. In this section, we'll try to make sense of what all those terms mean. After you are familiar with the terms, everything will start making more sense:

Node: Any machine or cloud instance that you are configuring using Chef is known as a node. On a Chef server it's an object comprising of attributes and a run list specific to the instance.
Chef server: A Chef server is a machine running chef-core-api, chef-solr, chef-web-ui, chef-expander, and chef-validator along with a backend data store such as PostGre/CouchDB and a messaging system such as RabbitMQ.
Workstation: This is the machine where we'll be writing our Chef code.
Repository: This could be a svn/Git repository where we'll be committing our code. This is useful to maintain revisions of code.
Knife: This is a tool that you can use to manage different aspects of Chef.
Cookbook: This is where you define anything and everything related to your infrastructure code. Cookbooks contain recipes, attributes, files/directories to be set up, templates, and so on.
Recipes: Theses are part of a cookbook and most of the code meat goes into recipes.
Attributes: Every code requires variables, and attributes are like variables holding values, which can be overridden.
Roles: These are a way of arranging cookbooks together. For example, a web server is a role and it can comprise of cookbooks to set up the Nginx web server along with OpenSSL and a few other things.
Run-list: This is an ordered list comprising of roles and/or recipes. The chef-client looks at items in run_list and executes them in an order specified in run_list.
Resources: The chef-client does multiple tasks such as setting up packages, creating users, setting up cron jobs, executing scripts, and so on. Since Chef is meant to be platform-agnostic, we don't use service providers explicit to the system to do these jobs. For example, we don't say yum installs this package, instead we use a resource provider called package, which internally decides which underlying system to choose for the job eventually. This is pretty useful as it helps keep Chef code agnostic to platform changes.
LWRP: Lightweight resources and providers (LWRP) are custom resources and providers that provide a way to perform a certain action. For example, you may write your own LWRP to manage Git repositories or install packages using Makefiles and so on.
Metadata: A metadata file describes properties of a cookbook such as version, dependencies, and so on, and it's used to verify that a cookbook is deployed correctly on a node.
Templates: Often, all we want to do is to specify a configuration that changes due to certain parameters, such as environment and so on. Templates allow for the creation of such configurations.
chef-client: This is an agent that will run on instances that we want to bootstrap using Chef.
Ohai: This is a piece of code that allows us to fetch useful information about a system along with other desired information. Ohai is used extensively to generate attributes that help in defining a node during a chef-client run.
DSL: Chef cookbooks are primarily written in Ruby. Chef provides a Domain Specific Language (DSL) that helps to write a code easily and quickly.
chef-solo: It's a tool similar to chef-client that will help us to execute a chef code.
chef-zero: It's a lightweight, in-memory implementation of the Chef server, which can be invoked on a node using chef-client –z. This is going to be a standard going forward and will be replacing chef-solo in the future.

Now that we know the language, let's jump into the world of Chef and see what happens when a chef-client run happens.

The anatomy of a Chef run

A Chef run here implies either the execution of chef-client or chef-solo, and we'll look at each of them separately.

A Chef run using chef-client

As we learned earlier while understanding terminology, a chef-client is an agent that runs on machines that are meant to be configured using Chef. The chef-client agent is meant to be executed in an environment where we are using Chef in a client-server architecture.

Upon the invocation of a chef-client, the following things happen:

Ohai is executed and automatic attributes are collected, which are eventually used to build a node object
Authentication with a chef-server
Synchronization of cookbooks
Loading of cookbooks and convergence
Checking for the status of chef-client run, reporting, and exception handling.

The chef-client, by default, looks for a configuration file named client.rb. On Linux/Unix-based machines this file is located at /etc/chef/client.rb. On Windows, this file is located at C:\chef\client.rb.

The chef-client command supports many options. The following option indicates which configuration file to use. By default, /etc/chef/client.rb is used for the purpose of a Chef run:

-c CONFIG, --config CONFIG

Tip

Downloading the example code

You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

The following option indicates that chef-client will be executed as a daemon and not as a foreground process. This option is only available on Linux/Unix. To run chef-client as a service in a Windows environment, use the chef-client::service recipe in the chef-client cookbook:

-d, --daemonize

The following option specifies the name of the environment:

-E ENVIRONMENT, --environment ENVIRONMENT

By default, a chef-client run forks a process where the cookbooks are executed. This helps prevent issues such as memory leaks and also helps to run a chef code with a steady amount of memory:

-f, --fork

The following option specifies the output format: summary (default), .json, .yaml, .txt, and .pp:

-F FORMAT, --format FORMAT

The following option indicates that the formatter output will be used instead of the logger output:

--force-formatter

The following option indicates that the logger output will be used instead of the formatter output:

--force-logger

The following option specifies a path to a JSON file, which will be used to override attributes and maybe specify run_list as well:

-j PATH, --json-attribute PATH

The following option specifies the location of a file containing a client key. The default location is /etc/chef/client.pem:

-k KEYFILE, --client KEYFILE

When a chef-client first registers a new machine with a chef-server, it doesn't have /etc/chef/client.pem. It contacts the chef-server with a key called validation_key (default location: /etc/chef/validation.pem). Upon contacting the chef-server, the chef-server responds with a new client key, which is stored in /etc/chef/client.pem. Going forward, every communication with a chef-server is authenticated with /etc/chef/client.pem:

-K KEYFILE, --validation_key KEYFILE

The following option is the name with which a machine is registered with a chef-server. The default name of the node is FQDN:

-N NODENAME, --node-name NODENAME

The following command replaces the current run list with specified items:

-o RUN_LIST_ITEM, --override-runlist RUN_LIST_ITEM

The following option provides a number in seconds to add an interval that determines how frequently a chef-client is executed. This option is useful when a chef-client is executed in daemon mode:

-s SECONDS, -splay SECONDS

The following command indicates that the chef-client executable will be run in the why-run mode. It's a dry-run mode where a chef-client run does everything, but it doesn't modify the system:

-W, --why-run

The following command specifies the location in which process identification number (PID) is saved. This is useful to manage a chef daemon via a process management system such as Monit:

-P PID_FILE, --pid PID_FILE

Let's presume we've already written a cookbook to install and configure a popular web server called Nginx.

We will create two files on our target machine:

client.rb: For our setup, the location will be /etc/chef/client.rb. It is a default configuration that will be used by a chef-client executable:
```
log_level        :info
log_location     "/var/log/chef.log"
chef_server_url  "http://chef-server:4000"
environment      "production"
```
As you can see, we've mentioned in our configuration that log_level is INFO, the log file is stored at /var/log/chef.log, chef-client will connect to a Chef server hosted at a machine accessible by the name chef-server, and finally we have our setup distributed across different environments and this machine is in the production environment.
roles.json: For our setup, the location will be /etc/chef/roles.json. This is a .json file that defines attributes, and a run_list which will be used to fetch the concerned cookbooks from a chef-server and the bootstrap machine;
```
{
  "run_list":["role[webserver]"],
  "app_user": "www-data",
  "log_dir": "/var/log",
}
```
As you can see, we've defined a run_list that comprises of a role called webserver. Along with this, we've specified two attributes: app_user and log_dir.

With client.rb and roles.json in place, now you can run chef-client as follows:

#chef-client –j /etc/chef/roles.json

The following image describes the steps as they happen during the chef-client run:

Let's look at each step closely.

Step 1 – Building a node object

As a first step, a chef-client will build the node object. To do this, the system is profiled first by Ohai.

Ohai returns a bunch of information about the system in a .json format. The following is an output from the Ohai run on our chef-eg01 instance:

# ohai
{
  "languages": {
    "ruby": {
      "platform": "x86_64-linux",
      "version": "2.1.0",
      "release_date": "2013-12-25",
      . . .
    },
    "python": {
      "version": "2.6.6",
      "builddate": "Jun 18 2012, 14:18:47"
    },
    "perl": {
      "version": "5.10.1",
      "archname": "x86_64-linux-thread-multi"
    },
    "lua": {
      "version": "5.1.4"
    },
    "java": {
      "version": "1.7.0_09",
      "runtime": {
        "name": "Java(TM) SE Runtime Environment",
        "build": "1.7.0_09-b05"
      },
      "hotspot": {
        "name": "Java HotSpot(TM) 64-Bit Server VM",
        "build": "23.5-b02, mixed mode"
      }
    }
  },
  "kernel": {
    "name": "Linux",
    "release": "2.6.32-220.23.1.el6.x86_64",
    "version": "#1 SMP Mon Jun 18 18:58:52 BST 2012",
    "machine": "x86_64",
   },
    "os": "GNU/Linux"
  },
  "os": "linux",
  "os_version": "2.6.32-220.23.1.el6.x86_64",
  "lsb": {
    "id": "CentOS",
    "description": "CentOS release 6.2 (Final)",
    "release": "6.2",
    "codename": "Final"
  },
  . . .
  "chef_packages": {
    "ohai": {
      "version": "6.14.0",
      "ohai_root": "/usr/local/rvm/gems/ruby-2.1.0/gems/ohai-6.14.0/lib/ohai"
    },
    "chef": {
      "version": "11.10.4",
      "chef_root": "/usr/local/rvm/gems/ruby-2.1.0/gems/chef-11.10.4/lib"
    }
  },
  "hostname": "chef-eg01",
  "fqdn": "chef-eg01.sychonet.com",
  "domain": "sychonet.com",
  "network": {
    "interfaces": {
      "lo": {
      . . .
      },
      "eth0": {
      . . .
      }
  },
  "ipaddress": "10.0.0.42",
  "macaddress": "0A:F8:4C:7A:C3:B2",
  "ohai_time": 1397945435.3669002,
  "dmi": {
    "dmidecode_version": "2.11"
  },
  "keys": {
    "ssh": {
      "host_dsa_public":"XXXXXXX",
      "host_rsa_public":"XXXXXXX
    }
  },
  . . .
}

As we can see, Ohai gave us plenty of useful information about our machine, such as the different language interpreters installed on the system, kernel version, OS platform and release, network, SSH keys, disks, RAM, and so on. All this information, that is automatic attributes, along with the node name, is used to build and register a node object with a chef-server. The default name of the node object is FQDN, as returned by Ohai. However, we can always override the node name in the client.rb configuration file.

Step 2 – Authenticate

We won't want our private chef-server to be responding to requests made by anyone. To accomplish this, each request to the Chef server is accompanied with some headers encrypted using the private key (client.pem).

As part of this step, a chef-client checks the presence of the /etc/chef/client.pem file, which is used for the purpose of authentication.

If no client.pem is present, a chef-client looks for a /etc/chef/validation.pem file, which is a private key assigned to the chef-validator. Once the chef-validator has authenticated itself to a chef-server, a chef-server creates a public/private key pair. The chef-server keeps a public key with itself, while a private key is sent back to a chef-client. After this step, our node object built in step 1 is registered with the chef-server.

Note

After the initial chef-client run is over, the chef-validator key is no longer required and can (ideally should) be deleted from the machine.

Step 3 – Synchronization of cookbooks

Now, since we are authenticated, we can go about fetching cookbooks from a chef-server. However, to send cookbooks to the relevant instance, a chef-server has to know which cookbooks to send across.

In this step, a chef-client fetches a node object from the chef-server. A node object defines what is in run_list and what attributes are associated with the node. A run_list list defines what cookbooks will be downloaded from a chef-server.

The following is what we have in our run_list:

"run_list":["role[webserver]"]

Our run_list comprises of one element called role[webserver]. A role is a way in which the Chef world organizes cookbooks together under one hood. Here is what our role looks like:

webserver.rb
# Role Name:: webserver
# Copyright 2014, Sychonet
# Author: maxc0d3r@sychonet.com

name "webserver"
description "This role configures nginx webserver"

run_list  "recipe[nginx]","recipe[base]"
override_attributes(
  :app => {
    :base => "/apps",
    :user => "ubuntu",
    :group => "ubuntu",
    :log => "/var/log/nginx",
    :data => "/data"
  }
)

Our role has run_list, which comprises of two elements: recipe[passenger-nginx] and recipe[base]. These recipes contain code that will be used to bootstrap a machine using Chef. Along with this, we've a few attributes:

node[:app][:base] = "/apps"
node[:app][:user] = "Ubuntu"
node[:app][:group] = "Ubuntu"
node[:app][:log] = "/var/log/nginx"
node[:app][:data] = "/data"

We will be using these attributes in our recipes to set up a machine according to our requirements. These attributes may already be defined in our cookbook and if they are, then they are overridden here.

Here is what a typical node json object looks like:

{
  "name": "chef-eg01.sychonet.com",
  "json_class": "Chef::Node",
  "chef_type": "node",
  "chef_environment": "production",
  "automatic": { . . . },
  "default": { . . . },
  "normal": { . . . },
  "override": { . . . },
  "run_list": [ . . . ]
}

Once the chef-client has obtained the node json object from the chef-server, it expands run_list. The run_list defined in a node object contains roles and recipes, and roles contain run_list that again contains further roles and recipes. During the execution of a chef-client, run_list gets expanded to the level of recipes.

Now, with a list of recipes to be executed on the machine, a chef-client downloads all the cookbooks mentioned in the expanded run_list from the chef server. Some cookbooks might not really be defined in run_list, but might be part of a dependency and those cookbooks are also downloaded as part of this event. A chef server maintains different versions of cookbooks and hence, if we want, we can request a specific version of a cookbook by specifying it as part of run_list, as follows:

{"run_list":["recipe[nginx@1.4.2]"]}

This will set up version 1.4.2 of the nginx recipe. We can also mention a version in the dependency or environment as follows:

depends "nginx", "= 1.4.2"

Alternatively, we can use the following code:

cookbook "nginx", "= 1.4.2"

Downloaded cookbooks are saved in a local filesystem on a machine at the location specified by file_cache_path, defined in client.rb (defaults to /var/chef/cache).

Upon subsequent chef-client runs, the cookbooks that haven't changed since the last run aren't downloaded and only the changed cookbooks are resynced.

Step 4 – Loading of cookbooks and convergence

Now, with all the cookbooks synchronized, a chef-client loads the components in the following order:

Libraries: Theses are loaded first so that all language extensions and Ruby classes are available.
Attributes: An attribute file updates node attributes and recipes.
Definitions: Theses must be loaded before recipes because they create new pseudo-resources.
Recipes: At this point, recipes are evaluated. Nothing is done with any resource defined in the recipe.

Recipes are loaded in the order they are specified in run_list. This is a very important concept to grasp because it can be a deal breaker if not understood properly. Let's look at our run_list in /etc/chef/roles.json:

"run_list":["role[webserver]"]

The webserver role in turn defines the following run_list:

run_list  "recipe[nginx]","recipe[base]"

This implies that the expanded run_list will look something like the following:

run_list  "recipe[nginx]","recipe[base]"

Now, if there are things mentioned in recipe[nginx] that require things that are being set up in recipe[base], then our Chef run will fail. For example, say we are setting up a user www-data in recipe[base] and we need Nginx to be started as a service with the user www-data in recipe[nginx], then it won't work because the www-data user won't be created until the base recipe is executed and it'll only be executed once recipe[nginx] has been executed.

At this point in time, all the evaluated resources found in recipes are put in resource collection, which is an array of each evaluated resource. Any external Ruby code is also executed at this point in time.

Now, with resource collection ready for use, a Chef run reaches a stage of execution.

Chef iterates through a resource collection in the following order:

It runs specified actions for each resource
A provider knows how to perform actions

Step 5 – Reporting and exception handling

Once a chef-client run has ended, the status of the run is checked. If there has been an error, Chef exits with unhandled exception and we can write exception handlers to handle such situations. For example, we might want to notify a system administrator about an issue with the chef-client run.

In the event of success as well, we might want to do certain things and this is handled via report handlers. For example, we might want to push a message to a queue saying that a machine has been bootstrapped successfully.

Using chef-solo

chef-solo is another executable that can be used to bootstrap any machine using cookbooks.

There are times when the need for a chef-server just isn't there, for example, when testing a newly written Chef cookbook on a virtual machine. During these times, we can't make use of a chef-client, as a chef-client requires a chef-server to communicate with.

The chef-solo allows using cookbooks with nodes without requiring a chef-server. It runs locally and requires those cookbooks (along with dependencies) to be present locally on the machine too.

Other than this difference, the chef-solo doesn't provide support for the following features:

Search
Authentication or authorization
Centralized distribution of cookbooks
Centralized API to interact with different infrastructure components.

The chef-solo can pick up cookbooks from either a local directory or URL where a tar.gz archive of the cookbook is present.

The chef-solo command uses the /etc/chef/solo.rb configuration file, or we can also specify an alternate path for this configuration file using the –config option during the chef-solo execution.

The chef-solo, by default, will look for data bags at /var/chef/data_bags. However, this location can be changed by specifying an alternate path in the data_bag_path attribute defined in solo.rb. The chef-solo picks up roles from the /var/chef/roles folder, but this location again can be modified by specifying an alternate path in the role_path attribute in solo.rb.

Other than the options supported by a chef-client, the chef-solo executable supports the following option:

-r RECIPE_URL, --recipe-url RECIPE_URL

A URL from where a remote cookbook's tar.gz will be downloaded.

For example:

#chef-solo –c ~/solo.rb –j ~/node.json –r http://repo.sychonet.com/chef-solo.tar.gz

The tar.gz file is first archived into file_cache_path and finally, extracted to cookbook_path.

Now that we understand how the Chef run happens, let's get our hands dirty and go about setting up our developer workstation.

Setting up a work environment

As we saw earlier, the Chef ecosystem comprises of three components: chef-server, chef-client, and a developer workstation.

We'll be developing all our beautiful Chef codes on our workstation. As we are developing a code, it's good practice to keep our code in some version control system such as git/svn/mercurial and so on. We'll choose Git for our purpose and I'll presume you've a repository called chef-repo that is being tracked by Git.

The following software should be installed on your machine before you try to set up your workstation:

Ruby (Preferably, 1.9.x).
We need Chef and Knife installed on our workstation and it's pretty easy to go about installing Chef along with Knife using the Ruby gems. Just open up a terminal and issue the command:
```
#gem install chef
```
Once Chef is installed, create a .chef folder in your home directory and create a knife.rb file in it.

Knife is a tool using which we'll use to communicate with a chef-server. Knife can be used for lots of purposes such as managing cookbooks, nodes, API clients, roles, environments, and so on. Knife also comes with plugins that allow it to be used for various other useful purposes. We'll learn more about them in later chapters.

Knife needs the knife.rb file present in the $HOME/.chef folder. The following is a sample knife.rb file:

log_level               :info
log_location            STDOUT
node_name               'NAME_OF_YOUR_CHOICE'
client_key              '~/.chef/NAME_OF_YOUR_CHOICE.pem'
validation_client_name  'chef-validator'
validation_key          '~/.chef/validation.pem'
chef_server_url         'http://chef-server.sychonet.com:4000'
cache_type              'BasicFile'
cache_options           (:path => '~/.chef/checksums')
cookbook_path           [ '~/code/chef-repo/cookbooks' ]

Connect to your chef-server web interface and visit the client section and create a new client with a name of your choice (ensure that no client with the same name exists on the chef-server):

Once you've created the client, a chef-server will respond with a public/private key pair as shown in the following screenshot:

Copy the contents of the private key and store them in ~/.chef/<NAME_OF_YOUR_CHOICE>.pem

Also, copy the private key for the chef-validator (/etc/chef/validation.pem) from the chef-server to ~/.chef/validation.pem.

Specify NAME_OF_YOUR_CHOICE as the node name.

As you can see, we've specified cookbook_path to be ~/code/chef-repo/cookbooks. I'm presuming that you'll be storing your Chef cookbooks inside this folder.

Create the following directory structure inside ~/code/chef-repo:

chef-repo
    ├── cookbooks
    ├── data_bags
    ├── environments
    └── roles

The cookbooks directory will hold our cookbooks, the data_bags directory will contain data bags, the environments directory will contain configuration files for different environments, and the roles directory will contain files associated with different roles.

Once you've created these directories, commit them to your Git repository.

Now, let's try to see if we are able to make use of the Knife executable and query the Chef server:

$knife client list
chef-validator
chef-webui
chef-eg01

This command will list all the available API clients registered with the chef-server. As you can see, chef-eg01 is a newly created client and it's now registered with the chef-server.

Knife caches the checksum of Ruby and ERB files when performing a cookbook syntax check with knife cookbook test or knife cookbook upload. The cache_type variable defines which type of cache to make use of. The most used type is BasicFile and it's probably best to leave it at that.

The cache_options is a hash for options related to caching. For BasicFile, :path should be the location on the filesystem where Knife has write access.

If you want the Knife cookbook to create a command to prefill values for copyright and e-mail in comments, you can also specify the following options in your knife.rb file:

cookbook_copyright "Company name"
cookbook_email "Email address"

With this setup, now we are ready to start creating new cookbooks, roles, and environments, and manage them along with nodes and clients using Knife from our workstation.

Before we jump into cookbook creation and other exciting stuff, we need to ensure that we follow a test-driven approach to our Chef development. We will make use of test-kitchen to help us write Chef cookbooks that are tested thoroughly before being pushed to a chef-server.

test-kitchen can be installed as a gem:

$ gem install test-kitchen

Also, download Vagrant from http://www.vagrantup.com and install it.

If you want some help, use the help option of the kitchen command:

$ kitchen help
Commands:
  kitchen console                         # Kitchen Console!
  kitchen converge [INSTANCE|REGEXP|all]  # Converge one or more instances
  kitchen create [INSTANCE|REGEXP|all]    # Create one or more instances
  kitchen destroy [INSTANCE|REGEXP|all]   # Destroy one or more instances
  kitchen diagnose [INSTANCE|REGEXP|all]  # Show computed diagnostic configuration
  kitchen driver                          # Driver subcommands
  kitchen driver create [NAME]            # Create a new Kitchen Driver gem project
  kitchen driver discover                 # Discover Test Kitchen drivers published on RubyGems
  kitchen driver help [COMMAND]           # Describe subcommands or one specific subcommand
  kitchen help [COMMAND]                  # Describe available commands or one specific command
  kitchen init                            # Adds some configuration to your cookbook so Kitchen can rock
  kitchen list [INSTANCE|REGEXP|all]      # Lists one or more instances
  kitchen login INSTANCE|REGEXP           # Log in to one instance
  kitchen setup [INSTANCE|REGEXP|all]     # Setup one or more instances
  kitchen test [INSTANCE|REGEXP|all]      # Test one or more instances
  kitchen verify [INSTANCE|REGEXP|all]    # Verify one or more instances
  kitchen version                         # Print Kitchen's version information

Now, let's create a new cookbook called passenger-nginx:

$knife cookbook create passenger-nginx

Now, we'll add test-kitchen to our project using the init subcommand:

$ kitchen init
create .kitchen.yml
create test/integration/default
run gem install kitchen-vagrant from "."
Fetching: kitchen-vagrant-0.14.0.gem (100%)
Successfully installed kitchen-vagrant-0.14.0
Parsing documentation for kitchen-vagrant-0.14.0
Installing ri documentation for kitchen-vagrant-0.14.0
Done installing documentation for kitchen-vagrant after 0 seconds
1 gem installed

The kitchen init command has created a configuration file called .kitchen.yml, along with a test/integration/default directory.

It also went on to install a gem called kitchen-vagrant. kitchen needs a virtual machine to test run the chef code, and drivers are responsible for managing virtual machines. By default, kitchen makes use of Vagrant to manage the virtual machine.

Let's see what we have in our configuration file, kitchen.yml:

$ cat .kitchen.yml
---
driver:
  name: vagrant
provisioner:
  name: chef_solo
platforms:
  - name: ubuntu-12.04
  - name: centos-6.4
suites:
  - name: default
    run_list:
      - recipe[cb-test1::default]
    attributes:

The file is divided into four sections:

Driver: This is where we set up basic stuff such as the SSH username and credentials. Under this section, we've a name property with a vagrant value. This tells kitchen to make use of the kitchen-vagrant driver.
Provisioner: This tells kitchen to make use of a chef-solo to apply the cookbook to a newly created virtual machine.
Platforms: This lists the operating systems on which we want to run our code.
Suites: Here we describe what we wish to test.

Now, let's see what we have on our hands:

$ kitchen list
Instance             Driver   Provisioner  Last Action
default-ubuntu-1204  Vagrant  ChefSolo     <Not Created>
default-centos-64    Vagrant  ChefSolo     <Not Created>

As you can see, it's listing two instances: default-ubuntu-1204 and default-centos-64. These names are a combination of the suite name and the platform name.

Now, let's spin up one instance to see what happens:

$ kitchen create default-ubuntu-1204
-----> Starting Kitchen (v1.2.1)
-----> Creating <default-ubuntu-1204>...
       Bringing machine 'default' up with 'virtualbox' provider...
       ==> default: Box 'opscode-ubuntu-12.04' could not be found. Attempting to find and install...
           default: Box Provider: virtualbox
           default: Box Version: >= 0
       ==> default: Adding box 'opscode-ubuntu-12.04' (v0) for provider: virtualbox
           default: Downloading: https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-12.04_chef-provisionerless.box
       ==> default: Successfully added box 'opscode-ubuntu-12.04' (v0) for 'virtualbox'!
       ==> default: Importing base box 'opscode-ubuntu-12.04'...
       ==> default: Matching MAC address for NAT networking...
       ==> default: Setting the name of the VM: default-ubuntu-1204_default_1398006642518_53572
       ==> default: Clearing any previously set network interfaces...
       ==> default: Preparing network interfaces based on configuration...
           default: Adapter 1: nat
       ==> default: Forwarding ports...
           default: 22 => 2222 (adapter 1)
       ==> default: Running 'pre-boot' VM customizations...
       ==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...           default: SSH address: 127.0.0.1:2222
           default: SSH username: vagrant
           default: SSH auth method: private key
           default: Warning: Connection timeout. Retrying...
       ==> default: Machine booted and ready!
       ==> default: Checking for guest additions in VM...
       ==> default: Setting hostname...
       Vagrant instance <default-ubuntu-1204> created.
       Finished creating <default-ubuntu-1204> (4m4.17s).
-----> Kitchen is finished. (4m4.71s)

So, this leads to the downloading of a virtual machine image for Ubuntu 12.04 and, eventually, the machine boots up. The default username for SSH connection is vagrant.

Let us check the status of our instance again:

$ kitchen list
Instance             Driver   Provisioner  Last Action
default-ubuntu-1204  Vagrant  ChefSolo     Created
default-centos-64    Vagrant  ChefSolo     <Not Created>

So, our Ubuntu instance is up and running. Now, let's add some meat to our recipe:

#
# Cookbook Name:: cb-test1
# Recipe:: default
#
# Copyright 2014, Sychonet
#
# All rights reserved - Do Not Redistribute
#

package "nginx"

log "Cool. So we have nginx installed"

So, now we've got our recipe ready, let's let test-kitchen run it in our instance now:

$ kitchen converge default-ubuntu-1204
-----> Starting Kitchen (v1.2.1)
-----> Converging <default-ubuntu-1204>...
       Preparing files for transfer
       Preparing current project directory as a cookbook
       Removing non-cookbook files before transfer
-----> Installing Chef Omnibus (true)
       downloading https://www.getchef.com/chef/install.sh
         to file /tmp/install.sh
       trying wget...
Downloading Chef  for ubuntu...      
downloading https://www.getchef.com/chef/metadata?v=&prerelease=false&nightlies=false&p=ubuntu&pv=12.04&m=x86_64      
  to file /tmp/install.sh.1144/metadata.txt      
trying wget...      
url  https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chef_11.12.2-1_amd64.deb      
md5  cedd8a2df60a706e51f58adf8441971b      
sha256  af53e7ef602be6228dcbf68298e2613d3f37eb061975992abc6cd2d318e4a0c0      
downloaded metadata file looks valid...      
downloading https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chef_11.12.2-1_amd64.deb      
  to file /tmp/install.sh.1144/chef_11.12.2-1_amd64.deb      
trying wget...      
Comparing checksum with sha256sum...      
Installing Chef       
installing with dpkg...      
Selecting previously unselected package chef.      
(Reading database ... 56035 files and directories currently installed.)      
Unpacking chef (from .../chef_11.12.2-1_amd64.deb) ...      
Setting up chef (11.12.2-1) ...      
Thank you for installing Chef!      
       Transfering files to <default-ubuntu-1204>
[2014-04-20T15:50:31+00:00] INFO: Forking chef instance to converge...      
[2014-04-20T15:50:31+00:00] WARN:       
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *       
SSL validation of HTTPS requests is disabled. HTTPS connections are still      
encrypted, but chef is not able to detect forged replies or man in the middle      
attacks.      
      
To fix this issue add an entry like this to your configuration file:      
      
```      
  # Verify all HTTPS connections (recommended)      
  ssl_verify_mode :verify_peer      
      
  # OR, Verify only connections to chef-server      
  verify_api_cert true      
       ```
      
       To check your SSL configuration, or troubleshoot errors, you can use the
       `knife ssl check` command like so:
      
       ```
         knife ssl check -c /tmp/kitchen/solo.rb
       ```
      
       * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
      
Starting Chef Client, version 11.12.2      
[2014-04-20T15:50:31+00:00] INFO: *** Chef 11.12.2 ***      
[2014-04-20T15:50:31+00:00] INFO: Chef-client pid: 1225      
[2014-04-20T15:50:39+00:00] INFO: Setting the run_list to ["recipe[cb-test1::default]"] from CLI options      
[2014-04-20T15:50:39+00:00] INFO: Run List is [recipe[cb-test1::default]]      
[2014-04-20T15:50:39+00:00] INFO: Run List expands to [cb-test1::default]      
[2014-04-20T15:50:39+00:00] INFO: Starting Chef Run for default-ubuntu-1204      
[2014-04-20T15:50:39+00:00] INFO: Running start handlers      
[2014-04-20T15:50:39+00:00] INFO: Start handlers complete.      
Compiling Cookbooks...      
Converging 2 resources      
Recipe: cb-test1::default      
  * package[nginx] action install[2014-04-20T15:50:39+00:00] INFO: Processing package[nginx] action install (cb-test1::default line 10)      
      
           - install version 1.1.19-1ubuntu0.6 of package nginx
      
  * log[Cool. So we have nginx installed] action write[2014-04-20T15:50:52+00:00] INFO: Processing log[Cool. So we have nginx installed] action write (cb-test1::default line 12)      
[2014-04-20T15:50:52+00:00] INFO: Cool. So we have nginx installed      
      
      
[2014-04-20T15:50:52+00:00] INFO: Chef Run complete in 12.923797655 seconds      
      
Running handlers:      
[2014-04-20T15:50:52+00:00] INFO: Running report handlers      
Running handlers complete      
      
[2014-04-20T15:50:52+00:00] INFO: Report handlers complete      
Chef Client finished, 2/2 resources updated in 21.14983058 seconds      
       Finished converging <default-ubuntu-1204> (2m10.10s).
-----> Kitchen is finished. (2m10.41s)

So, here is what happened under the hood when kitchen converge was executed:

Chef was installed on an Ubuntu instance
Our cb-test1 cookbook and a chef-solo configuration were uploaded to an Ubuntu instance.
The Chef run was initiated using run_list and attributes defined in .kitchen.yml

If the exit code of the kitchen command is 0, then the command run was successful. If it's not 0, then any part of the operation associated with the command was not successful.

Let's check the status of our instance once more:

$ kitchen list
Instance             Driver   Provisioner  Last Action
default-ubuntu-1204  Vagrant  ChefSolo     Converged
default-centos-64    Vagrant  ChefSolo     <Not Created>

So, our instance is converged, but we still don't know if nginx was installed successfully or not. One way to check this is to log in to the instance using the following command:

$ kitchen login default-ubuntu-1204

Once you've logged in to the system, you can now go ahead and check for the presence of the binary named nginx:

vagrant@default-ubuntu-1204:~$ which nginx
/usr/sbin/nginx

So, Nginx is indeed installed.

However, with kitchen, we no longer need to take the pain of logging in to the system and verifying the installation. We can do this by writing a test case.

We'll make use of bash automated testing system (bats), called for this purpose.

Create a directory using the following command:

$ mkdir -p test/integration/default/bats

Create a new file package test.bats under the bats directory:

#!/usr/bin/env bats

@test "nginx binary is found in PATH"
{
  run which nginx
  [ "$status" -eq 0 ]
}

Now, let's run our test using kitchen verify:

$ kitchen verify default-ubuntu-1204
-----> Starting Kitchen (v1.2.1)
-----> Setting up <default-ubuntu-1204>...
Fetching: thor-0.19.0.gem (100%)      
Fetching: busser-0.6.2.gem (100%)      
Successfully installed thor-0.19.0      
Successfully installed busser-0.6.2      
2 gems installed      
-----> Setting up Busser      
       Creating BUSSER_ROOT in /tmp/busser      
       Creating busser binstub      
       Plugin bats installed (version 0.2.0)      
-----> Running postinstall for bats plugin      
Installed Bats to /tmp/busser/vendor/bats/bin/bats      
       Finished setting up <default-ubuntu-1204> (1m41.31s).
-----> Verifying <default-ubuntu-1204>...
       Suite path directory /tmp/busser/suites does not exist, skipping.      
Uploading /tmp/busser/suites/bats/package-test.bats (mode=0644)      
-----> Running bats test suite
✓ nginx binary is found in PATH      
      
1 test, 0 failures      
       Finished verifying <default-ubuntu-1204> (0m1.03s).
-----> Kitchen is finished. (0m1.51s)

So, we see that our test has successfully passed verification, and we can proudly go ahead and upload our cookbook to the chef-server and trigger a chef-client run on the concerned instance.

jeevan dongre Sep 08, 2015

Mr.Mayank Joshi has done a brilliant job.I suggest this to all the beginners and also experts who are working on chef or keen to take up a career on chef configuration management tool. Everything is patiently written, focused on even the small details of the platform.

Amazon Verified review

Amazon Customer May 23, 2016

The book is a good introduction for Chef.

Jason and Ashley Morgan Jan 22, 2016

I'm still in the process of reading it so obviously I'll need to post an update when I either finish or give up. This is nowhere near well written and the author seems to jump around and assume the reader already has a deep understanding of chef. This is my second chef book, I've done multiple training sessions, and I'm responsible for maintaining chef and writing recipes for my production environment. I bought this book to give me a deep dive into chef. I think that's what it's providing but it's hard to tell. The author jumps around rapidly and continually references portions of the product that he hasn't covered yet.I'm still holding out hope that decyphering the authors intent will get me to a much deeper understanding of chef but I haven't found it helpful so far. This is definitely not for beginners. That all being said the author clearly understands chef and I've already come across multiple explanations that gave me insight into aspects of chef that I didn't understand before. It's just really frustrating trying to dig them out of the poorly written and scattered text.

Manuel García del Ojo Jul 25, 2015

I am very disapointed with this book. I expected a walkthrough explaining how to build a project in the first chapters but the only thing it explains is the chef resources, elements and so on...To sum up, if you already know to how start a project is the right book for you, but not if you are a beginner with chef like me.

Mastering Chef: Build, deploy, and manage your IT infrastructure to deliver a successful automated system with Chef in any environment

What do you get with eBook?

Mastering Chef

Chapter 1. Introduction to the Chef Ecosystem

Different modes of running Chef

Terminology used in the world of Chef

The anatomy of a Chef run

A Chef run using chef-client

Tip

Step 1 – Building a node object

Step 2 – Authenticate

Note

Step 3 – Synchronization of cookbooks

Step 4 – Loading of cookbooks and convergence

Step 5 – Reporting and exception handling

Using chef-solo

Setting up a work environment

Summary

Page 1 of 6

Description

Who is this book for?

What you will learn

Product Details

What do you get with eBook?

Product Details

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

People who bought this also bought

About the author

FAQs

Mastering Chef: Build, deploy, and manage your IT infrastructure to deliver a successful automated system with Chef in any environment

What do you get with eBook?

Contact Details

Billing Address

Step 1 – Building a node object

Step 2 – Authenticate

Step 3 – Synchronization of cookbooks

Step 4 – Loading of cookbooks and convergence

Step 5 – Reporting and exception handling

Description

Who is this book for?

What you will learn

Product Details

What do you get with eBook?

Contact Details

Billing Address

Product Details

Packt Subscriptions

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

People who bought this also bought

About the author

FAQs