Chapter 1. Chef Infrastructure

"What made Manhattan Manhattan was the underground infrastructure, that engineering marvel."
Andrew Cuomo

A well-engineered infrastructure builds the basis for successful companies. In this chapter, we will see how to set up an infrastructure around Chef as the basis of your infrastructure as code. We'll cover the following recipes in this chapter:

Using version control
Installing the Chef Development Kit on your workstation
Using the hosted Chef platform
Managing virtual machines with Vagrant
Creating and using cookbooks
Inspecting files on your Chef server with knife
Defining cookbook dependencies
Managing cookbook dependencies with Berkshelf
Using custom knife plugins
Deleting a node from the Chef server
Developing recipes with local mode
Using roles
Using environments
Freezing cookbooks
Running the Chef client as a daemon

Introduction

This chapter will cover the basics of Chef, including common terminology, workflow practices, and various tools that work with Chef. We will explore version control using Git, walk through working with community cookbooks, and run those cookbooks on your own servers.

First, let's talk about some important terms used in the Chef universe.

A cookbook is a collection of all the components needed to change something on a server. Things such as installing MySQL or configuring SSH can be done by cookbooks. The most important parts of cookbooks are recipes, which tell Chef which resources you want to configure on your host.

You need to deploy cookbooks to the nodes that you want to change. Chef offers multiple methods for this task. Most probably, you'll use a central Chef server. You can either run your own server or sign up for hosted Chef.

The Chef server is the central registry, where each node needs to be registered. The Chef server distributes the cookbooks you uploaded to it, to your nodes.

Knife is Chef's command-line tool to interact with the Chef server. You run it on your local workstation and use it to upload cookbooks and manage other aspects of Chef.

On your nodes, you need to install Chef Client—the program that runs on your nodes, retrieving cookbooks from the Chef server and executing them on the node.

In this chapter, we'll see the basic infrastructure components of your Chef setup at work and learn how to use the basic tools. Let's get started by looking at how to use Git as a version control system for your cookbooks.

Using version control

Do you manually back up every file before you change it? And do you invent creative file name extensions such as _me and _you when you try to collaborate on a file? If you answer yes to any of these, it's time to rethink your processes.

A version control system (VCS) helps you stay sane when dealing with important files and collaborating on them.

Using version control is a fundamental part of any infrastructure automation. There are multiple solutions to manage source version control, including Git, SVN, Mercurial, and Perforce. Due to its popularity among the Chef community, we will be using Git. However, you could easily use any other version control system with Chef.

Note

Don't even think about building your infrastructure as code without using a version control system to manage it!

Getting ready

You'll need Git installed on your local workstation. Either use your operating system's package manager (such as Apt on Ubuntu or Homebrew on OS X, or simply download the installer from www.git-scm.org.

Git is a distributed version control system. This means that you don't necessarily need a central host to store your repositories. However, in practice, using GitHub as your central repository has proven to be very helpful. In this book, I'll assume that you're using GitHub. Therefore, you need to go to www.github.com and create an account (which is free) to follow the instructions given in this book. Make sure that you upload your Secure Shell (SSH) key by following the instructions at https://help.github.com/articles/generating-ssh-keys, so that you're able to use the SSH protocol to interact with your GitHub account.

As soon as you have created your GitHub account, you should create your repository by visiting https://github.com and using chef-repo as the repository name.

Make sure you have wget installed on your local workstation, to be able to download the required files from public servers.

How to do it…

Before you can write any cookbooks, you need to set up your initial Git repository on your development box. Chef Software, Inc. provides an empty Chef repository to get you started. Let's see how you can set up your own Chef repository with Git, using Chef's skeleton:

Download Chef's skeleton repository as a tarball:

mma@laptop $ wget http://github.com/chef/chef-repo/tarball/master
...TRUNCATED OUTPUT...
2016-09-28 20:54:41 (9.26 MB/s) - 'master' saved [7332/7332]

Extract the downloaded tarball:
```
mma@laptop $ tar xzvf master
```

Rename the directory:

mma@laptop:~ $ mv chef-boneyard-chef-repo-* chef-repo

Change to your newly created Chef repository:
```
mma@laptop:~ $ cd chef-repo/
```

Initialize a fresh Git repository:

git init .
Initialized empty Git repository in /Users/mma/work/chef-repo/.git/

Connect your local repository to your remote repository on github.com. Make sure to replace mmarschall with your own GitHub username:
```
mma@laptop:~/chef-repo $ git remote add origin git@github.com:mmarschall/chef-repo.git
```

Configure Git with your user name and e-mail address:

mma@laptop:~/chef-repo $ git config --global user.email "you@example.com"
mma@laptop:~/chef-repo $ git config --global user.name "Your Name"

Add and commit Chef's default directory structure:

mma@laptop:~/chef-repo $ git add .
mma@laptop:~/chef-repo $ git commit -m "initial commit"
[master (root-commit) 6148b20] initial commit
 11 files changed, 545 insertions(+), 0 deletions(-)
 create mode 100644 .gitignore
...TRUNCATED OUTPUT...
create mode 100644 roles/README.md

Push your initialized repository to GitHub. This makes it available to all your co-workers to collaborate on:

mma@laptop:~/chef-repo $ git push -u origin master
...TRUNCATED OUTPUT...
To git@github.com:mmarschall/chef-repo.git
 * [new branch]      master -> master

How it works…

You have downloaded a tarball containing Chef's skeleton repository. Then, you initialized chef-repo and connected it to your own repository on GitHub.

After that, you added all the files from the tarball to your repository and committed them. This makes Git track your files and the changes you make later.

Finally, you pushed your repository to GitHub, so that your co-workers can use your code too.

There's more...

Let's assume you're working on the same chef-repo repository, together with your co-workers. They cloned the repository, added a new cookbook called other_cookbook, committed their changes locally, and pushed to GitHub. Now, it's time for you to get the new cookbook downloaded to your own laptop.

Pull your co-workers' changes from GitHub. This will merge their changes into your local copy of the repository. Use the pull subcommand:

mma@laptop:~/chef-repo $ git pull --rebase
From github.com:mmarschall/chef-repo
 * branch            master     -> FETCH_HEAD
...TRUNCATED OUTPUT...
create mode 100644 cookbooks/other_cookbook/recipes/default.rb

In the event of any conflicting changes, Git will help you merge and resolve them.

Installing the Chef Development Kit on your workstation

If you want to use Chef, you'll need to install the Chef Development Kit (DK) on your local workstation first. You'll have to develop your configurations locally and use Chef to distribute them to your Chef server.

Chef provides a fully packaged version, which does not have any external prerequisites. This fully packaged Chef is called the omnibus installer. We'll see how to use it in this section.

How to do it…

Let's see how to install the Chef DK on your local workstation using Chef's omnibus installer:

Download the Chef DK for your specific workstation platform from https://downloads.chef.io/chef-dk/ and run the installer.

Verify that Chef installed all the required components:

mma@laptop:~ $ chef verify
...TRUNCATED OUTPUT...
Verification of component 'test-kitchen' succeeded.
Verification of component 'chefspec' succeeded.
Verification of component 'rubocop' succeeded.
Verification of component 'knife-spork' succeeded.
Verification of component 'openssl' succeeded.
Verification of component 'delivery-cli' succeeded.
Verification of component 'opscode-pushy-client' succeeded.
Verification of component 'berkshelf' succeeded.
Verification of component 'chef-dk' succeeded.
Verification of component 'fauxhai' succeeded.
Verification of component 'inspec' succeeded.
Verification of component 'chef-sugar' succeeded.
Verification of component 'tk-policyfile-provisioner' succeeded.
Verification of component 'chef-provisioning' succeeded.
Verification of component 'kitchen-vagrant' succeeded.
Verification of component 'git' succeeded.
Verification of component 'chef-client' succeeded.
Verification of component 'generated-cookbooks-pass-chefspec' succeeded.
Verification of component 'package installation' succeeded.

Add the newly installed Ruby to your path:
```
mma@laptop:~ $ echo 'export PATH="/opt/chefdk/bin:/opt/chefdk/embedded/bin:$PATH"' >> ~/.bash_profile && source ~/.bash_profile
```
Note
You may not want to use (and don't have to use) ChefDK's Ruby, especially if you are a Rails Developer. If you're happily using your Ruby rvm or rbenv environment, you can continue to do so. Just ensure that ChefDK-provided applications appear first in your PATH, before any gem-installed versions, and you're good to go.
```
.chef/encrypted_data_bag_secret
```

How it works…

The omnibus installer will download Ruby and all required Ruby gems into /opt/chefdk.

Using the hosted Chef platform

If you want to get started with Chef right away (without the need to install your own Chef server) or want a third party to give you a Service Level Agreement (SLA) for your Chef server, you can sign up for hosted Chef by Chef Software, Inc. Chef Software, Inc. operates Chef as a cloud service. It's quick to set up and gives you full control, using users and groups to control access permissions to your Chef setup. We'll configure knife, Chef's command-line tool, to interact with hosted Chef, so that you can start managing your nodes.

Getting ready

Before being able to use hosted Chef, you need to sign up for the service. There is a free account for up to five nodes.

Visit http://manage.chef.io/signup and register for a free account.

I registered as the user webops with an organization short name of awo. An organization is the top-level entity for role-based access control in the Chef server.

After registering your account, it is time to prepare your organization to be used with your chef-repo repository.

How to do it…

Carry out the following steps to interact with the hosted Chef:

Create the configuration directory for your Chef client on your local workstation:
```
mma@laptop:~ $ cd ~/chef-repo
 mkdir .chef
```
Generate the knife config and put the downloaded knife.rb into the .chef directory inside your chef-repo directory. Make sure you have your user's private key saved as .chef/<YOUR USERNAME>.pem, (in my case it is .chef/webops.pem). If needed, you can reset it at https://id.chef.io/id/profile. Replace webops with the username you chose for hosted Chef, and awo with the short name you chose for your organization in your knife.rb file:
```
current_dir = File.dirname(__FILE__)
log_level                :info
log_location             STDOUT
node_name                "webops"
client_key               "#{current_dir}/webops.pem"
chef_server_url          "https://api.chef.io/organizations/awo"
cache_type               'BasicFile'
cache_options( :path => "#{ENV['HOME']}/.chef/checksums" )
cookbook_path            ["#{current_dir}/../cookbooks"]
```
Note
You should add the following code to your .gitingore file inside chef-repo to avoid your credentials ending up in your Git repository:
```
.chef/*.pem
```
Use knife to verify that you can connect to your hosted Chef organization. It should not have any clients, so far:
```
mma@laptop:~/chef-repo $ knife client list
```

How it works…

The following line of code in your knife.rb file tells knife where to find your user's private key. It is used to authenticate you with the Chef server:

client_key               "#{current_dir}/webops.pem"

Also, the following line of code in your knife.rb file tells knife that you are using hosted Chef. You will find your organization name as the last part of the URL:

chef_server_url          "https://api.chef.io/organizations/awo"

Using the knife.rb file and your user's key, you can now connect to your organization hosted by Chef Software, Inc.

There's more…

This setup is good for you if you do not want to worry about running, scaling, and updating your own Chef server and if you're happy with saving all your configuration data in the Cloud (under the control of Chef Software, Inc.).

Note

If you need to have all your configuration data within your own network boundaries, you can install Chef server on premises by choosing ON PREMISES CHEF at https://www.chef.io/chef/choose-your-version/ or install the Open Source version of Chef server directly from GitHub at https://github.com/chef/chef.

Managing virtual machines with Vagrant

Vagrant is a command-line tool that provides you with a configurable, reproducible, and portable development environment using VMs. It lets you define and use preconfigured disk images to create new VMs from. Also, you can configure Vagrant to use provisioners such as Shell scripts, Puppet, or Chef to bring your VM into the desired state.

Tip

Chef comes with Test Kitchen, which enables you to test your cookbooks on Vagrant without you needing to setup anything manually.

You only need to follow this section, if you want to learn how to use Vagrant and Chef for more advanced cases.

In this recipe, we will see how to use Vagrant to manage VMs using VirtualBox and Chef client as the provisioner.

Getting ready

Download and install VirtualBox at https://www.virtualbox.org/wiki/Downloads.
Download and install Vagrant at https://www.vagrantup.com/downloads.html.

Install the Omnibus Vagrant plugin to enable Vagrant to install the Chef client on your VM by running the following command:

mma@laptop:~/chef-repo $ vagrant plugin install vagrant-omnibus
Installing the 'vagrant-omnibus' plugin. This can take a few minutes...
Installed the plugin 'vagrant-omnibus (1.5.0)'!

How to do it…

Let's create and boot a virtual node by using Vagrant:

Visit https://github.com/chef/bento and choose a Vagrant box to base your VMs on. We'll use the amd64 image of ubuntu-16.04 in this example.
The URL of that box is http://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-16.04_chef-provisionerless.box.

Create a new Vagrantfile. Make sure that you replace <YOUR-ORG> with the name of your organization on the Chef server. Use the name and URL of the box file you noted down in the first step as config.vm.box and config.vm.box_url:

mma@laptop:~/chef-repo $ subl Vagrantfile
Vagrant.configure("2") do |config|
  config.vm.box = "opscode-ubuntu-16.04"
  config.vm.box_url = "http://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-16.04_chef-provisionerless.box"
  config.omnibus.chef_version = :latest

  config.vm.provision :chef_client do |chef|
    chef.provisioning_path = "/etc/chef"
    chef.chef_server_url = "https://api.chef.io/organizations/<YOUR_ORG>"
    chef.validation_key_path = ".chef/<YOUR_USER>.pem"
    chef.validation_client_name = "<YOUR_USER> "
    chef.node_name = "server"
  end
end

Create your virtual node using Vagrant:

mma@laptop:~/chef-repo $ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'opscode-ubuntu-16.04' could not be found. Attempting to find and install...
...TRUNCATED OUTPUT...
==> default: Importing base box 'opscode-ubuntu-16.04'...
...TRUNCATED OUTPUT...
==> default: Installing Chef latest Omnibus package...
...TRUNCATED OUTPUT...
==> default: Running chef-client...
==> default: Starting Chef Client, version 12.14.89
...TRUNCATED OUTPUT...

Log in to your virtual node using SSH:

mma@laptop:~/chef-repo $ vagrant ssh
Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-31-generic x86_64)
...TRUNCATED OUTPUT...
vagrant@server:~$

Log out of your virtual node:

vagrant@server:~$ exit
logout
Connection to 127.0.0.1 closed.
mma@laptop:~/chef-repo $

Validate that the Chef server knows your new virtual machine as a client called server:
```
mma@laptop:~/chef-repo $ knife client list
awo-validator
server
```
Go to https://manage.chef.io/organizations/<YOUR ORGANIZATION>/nodes and validate that your new virtual machine shows up as a registered node:

How it works…

The Vagrantfile is written in a Ruby Domain Specific Language (DSL) to configure the Vagrant virtual machines. We want to boot a simple Ubuntu VM. Let's go through the Vagrantfile step by step.

First, we create a config object. Vagrant will use this config object to configure the VM:

Vagrant.configure("2") do |config|
...
end

Inside the config block, we tell Vagrant which VM image to use, in order to boot the node:

config.vm.box = "opscode-ubuntu-16.04"
config.vm.box_url = "http://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-16.04_chef-provisionerless.box"

We want to boot our VM using a so-called Bento Box, provided by Chef. We use Ubuntu Version 16.04 here.

Note

If you have never used the box before, Vagrant will download the image file (a few hundred megabytes) when you run vagrant up for the first time.

As we want our VM to have the Chef client installed, we tell the omnibus vagrant plugin to use the latest version of the Chef client:

  config.omnibus.chef_version = :latest

After selecting the VM image to boot, we configure how to provision the box by using Chef. The Chef configuration happens in a nested Ruby block:

  config.vm.provision :chef_client do |chef|
  ...
  end

Inside this chef block, we need to instruct Vagrant on how to hook up our virtual node to the Chef server. First, we need to tell Vagrant where to store all the Chef stuff on your node:

    chef.provisioning_path = "/etc/chef"

Vagrant needs to know the API endpoint of your Chef server. If you use hosted Chef, it is https://api.chef.io/organizations/<YOUR_ORG>. You need to replace <YOUR_ORG> with the name of the organization that you created in your account on hosted Chef. If you are using your own Chef server, change the URL accordingly:

    chef.chef_server_url = "https://api.chef.io/organizations/<YOUR_ORG>"

While creating your user on hosted Chef, you must have downloaded your private key. Tell Vagrant where to find this file:

    chef.validation_key_path = ".chef/<YOUR_USER>.pem"

Also, you need to tell Vagrant which client it should use to validate itself against in the Chef server:

    chef.validation_client_name = "<YOUR_USER>"

Finally, you should tell Vagrant how to name your node:

    chef.node_name = "server"

After configuring your Vagrantfile, all you need to do is run the basic Vagrant commands such as vagrant up, vagrant provision, and vagrant ssh. To stop your VM, just run the vagrant halt command.

There's more…

If you want to start from scratch again, you will have to destroy your VM and delete both the client and the node from your Chef server by running the following command:

mma@laptop:~/chef-repo $ vagrant destroy
mma@laptop:~/chef-repo $ knife node delete server -y && knife client delete server -y

Alternatively, you may use the Vagrant Butcher plugin found at https://github.com/cassianoleal/vagrant-butcher.

Tip

Don't blindly trust Vagrant boxes downloaded from the Web; you never know what they contain.

Creating and using cookbooks

Cookbooks are an essential part of Chef. Basically, you describe the configurations you want to apply to your nodes in cookbooks. You can create them using the Chef executable installed by the Chef DK.

In this recipe, we'll create and apply a simple cookbook using the chef and knife command-line tools.

Getting ready

Make sure you have Chef DK installed and a node available for testing. Check out the installation instructions at http://learn.chef.io if you need help here.

Edit your knife.rb file (usually found in the hidden .chef directory) and add the following three lines to it, filling in your own values:

cookbook_copyright "your company"
cookbook_license "apachev2"
cookbook_email "your email address"

Note

The Apache 2 license is the most commonly license found in cookbooks, but you're free to choose whichever suits your needs. If you put none as cookbook_license, knife will put All rights reserved into your recipe's metadata file.

Chef will use the preceding values as the defaults whenever you create a new cookbook. We assume that you have a node called server registered with your Chef server, as described in the Managing virtual machines with Vagrant section in this chapter.

How to do it…

Carry out the following steps to create and use cookbooks:

Create a cookbook named my_cookbook by running the following command:

mma@laptop:~/chef-repo $ chef generate cookbook cookbooks/my_cookbook
Generating cookbook my_cookbook
- Ensuring correct cookbook file content
- Ensuring delivery configuration
- Ensuring correct delivery build cookbook content

Your cookbook is ready. Type `cd cookbooks/my_cookbook` to enter it.
...TRUNCATED OUTPUT...

Upload your new cookbook on the Chef server:

mma@laptop:~/chef-repo $ knife cookbook upload my_cookbook
Uploading my_cookbook    [0.1.0]
Uploaded 1 cookbook.

Add the cookbook to your node's run list. In this example, the name of the node is server:

mma@laptop:~/chef-repo $ knife node run_list add server 'recipe[my_cookbook]'
server:
  run_list: recipe[my_cookbook]

Run the Chef client on your node:
```
user@server:~$ sudo chef-client
```

Tip

If you're using a Vagrant VM as your server, you need to make sure to run vagrant up and vagrant ssh to be able to execute the Chef client on the node.

How it works…

The chef executable helps you to manage your local Chef Development environment. We used it here to generate the cookbook.

Knife is the command-line interface for the Chef server. It uses the RESTful API exposed by the Chef server to do its work and helps you to interact with the Chef server.

The knife command supports a host of commands structured as follows:

knife <subject> <command>

The <subject> used in this section is either cookbook or node. The commands we use are upload for the cookbook, and run_list add for the node.

There's more…

Before uploading your cookbook to the Chef server, it's a good idea to run it in Test Kitchen first. Test Kitchen will spin up a virtual machine, execute your cookbook, and destroy the virtual machine again. That way you can evaluate what your cookbook does before you upload it to the Chef server and run it on real nodes.

To run your cookbook with Test Kitchen on an Ubuntu 16.04 virtual machine, execute the following steps:

Create a configuration file for Test Kitchen for executing the default recipe of my_cookbook:

mma@laptop:~/chef-repo $ subl .kitchen.yml
---
driver:
  name: vagrant

provisioner:
  name: chef_zero

platforms:
  - name: ubuntu-16.04

suites:
  - name: default
    run_list:
      - recipe[my_cookbook::default]
 attributes:

Run kitchen test to execute the default recipe of my_cookbook:

mma@laptop:~/chef-repo $ kitchen test
-----> Starting Kitchen (v1.13.2)
...TRUNCATED OUTPUT...
-----> Kitchen is finished. (0m45.42s)

Inspecting files on your Chef server with knife

Sometimes, you may want to peek into the files stored on your Chef server. You might not be sure about an implementation detail of the specific cookbook version currently installed on your Chef server, and need to look it up. Knife can help you out by letting you show various aspects of the files stored on your Chef server.

Getting ready

Install the iptables community cookbook by executing the following command:
```
mma@laptop:~/chef-repo $ knife cookbook site install iptables
Installing iptables to /Users/mma/work/chef-repo/cookbooks
...TRUNCATED OUTPUT...
```
Note
Take a look at the following error:
```
ERROR: IOError: Cannot open or read ../chef-repo/cookbooks/iptables/metadata.rb!
```
If you get the preceding error, your cookbook only has a metadata.json file. Make sure that you delete it and create a valid metadata.rb, file instead.

Upload the iptables cookbook on your Chef server by executing the following command:

mma@laptop:~/chef-repo $ knife cookbook upload iptables --include-dependencies
Uploading iptables       [3.0.0]
Uploading compat_resource [12.14.7]
Uploaded 2 cookbooks.