Chef is a ruby framework, and its main purpose is to facilitate the automation, reuse, and documentation of server configurations. Chef allows you to treat server configuration just like any other type of software code.

There are three major components of this framework for almost all corporate-level Chef deployments—the workstation, the Chef Server, and the nodes. Let's talk a little about each of these components:

The Chef client is the tool that is deployed to all nodes and used to configure or reconfigure the node to its desired state. The Chef client is what executes the automation. It is the magician that, through the magic of Chef, transforms an ordinary server into the desired state node that the developer defined back on his workstation.

Ohai is the tool that gathers information about a node. Information such as platform details, operating system data, and processor information is made available to the Chef client so that the latter can have the know-how to bring the node to the desired state. Ohai is executed at the beginning of a Chef client run to gather the state of the node. At the end of a Chef client run, all of the data gathered by Ohai (usually) is shared as node data with the Chef server, and the shared data is available for searches done against the Chef server. Ohai can be extended via plugins, and we'll take a look at that in a later chapter.

Recipes are the building blocks used to define the desired states. Recipes are files of ruby code that define the commands to be run on nodes. They are like blueprints used to "build" a node. Recipes are collections of Chef resources. We will explain more about Chef resources shortly.

Cookbooks are collections of related recipes, templates, files, and custom resources. They provide organization and versioning for recipes. Each unique version of a cookbook represents unique sets of functionality, such as bug fixes or added features.

Cookbooks define a scenario, such as everything needed to install and configure apt-docker or Sublime text, and they contain all the elements needed to support the defined scenario.

Recipes and cookbooks provide modularity and let you easily reuse code.

A "run list" is, as the name would suggest, a list of, and the sequence for, the recipes, cookbooks, and policies (spoiler alert!) to be applied to a node. A run list contains all of the information required to configure a node to a desired state. That is, a Chef run list describes the desired final state of the node. It is important to note that, if the node's state already matches what the resources in the run list describe, then no action will be taken to change the node's state.

Roles are functional groupings of recipes and cookbooks used to describe the full blueprint needed for a node to become everything it is intended to be. Roles are reusable configurations, and they can be applied to multiple nodes to make functionally identical servers, such as a farm of web servers.

Chef resources are statements of configuration policy. They are defined in recipes and take actions through the Chef client to put the node into the desired state. Chef resources have some types: Package, Template, Service, and so on. They have a name and parameters. Also, Chef resources can send notifications to other resources.

Chef resources define what we want to happen to a node. They don't say how to do it. The how to do it is left to the providers which are platform-specific. That is to say, the way you install a package will be different depending on the OS, and the provider determines the correct way to do it—the "how". The Chef resource simply defines the "what," such as "install ntp".

Apart from the three types of Chef resources we saw earlier, let's look at some other important aspects of recipes and cookbooks.

Chef resources are idempotent. That is, applying them more than once results in the same outcome every time. If no inputs related to the resource have changed, then applying that resource won't change anything after the first application. In fact, if none of the inputs have changed, the corresponding commands don't even get run.

This is actually one of the most important concepts and features of Chef.

Data bags are containers for information that is not tied to a specific node. Data bags are the global variables of a Chef server. They can be used in recipes and can be searched like node data via the Chef server. One common use case for data bags is for user and group information. Items in a data bag can be encrypted. This allows secret information to be stored in them, for example, passwords.

Environments allow you to define specific cookbook versions that are applied to a given set of nodes. They permit you to model the stages of your infrastructure workflow, that is, Development, Test, Stage, and Production. By identifying specific nodes as Development, and other nodes as Production, for example, you can apply different versions of your cookbooks to the nodes based on their environment membership.

The supermarket is a site that provides shared cookbooks. There is a public supermarket that contains community-created and -maintained cookbooks. This site is hosted by Chef and is available at https://supermarket.chef.io. In addition to the public supermarket, anyone can create and manage a private supermarket that can host cookbooks intended for the private consumption of you and your organization.

The ChefDK contains everything you need to start working with Chef on a workstation. It provides all the tools that a developer needs to create and modify cookbooks and upload them to a Chef server. We are going to go into a lot of detail regarding the ChefDK later in this chapter.

You can jump ahead to the ChefDK sections now, or you can read on and learn how to set up your very own private Chef server.

Many companies will have strong reasons to keep their infrastructure configuration data within the firewalls of their datacenters, and Chef server has a mode for that. It's called on-premise or private Chef server.

There are some real benefits to using an in-house solution, including control. With a private installation, you have full control of your system, and for some that reason is enough. But there are additional considerations such as performance. Since a private Chef server will likely be physically closer to the node's network, Chef client runs will be faster. For example, if Chef is used in a load-based on-demand server deployment solution, then the extra speed the private Chef server can provide when converging your new Nodes can make all the difference in meeting the load demand in time.

With all the reasons to use an on-premise Chef server, there is a downside: you are responsible for the deployment, configuration, and maintenance of your Chef servers.

To get you started, let's take a look at the deployment and initial configuration of a standalone private Chef server now. We'll go through the steps to get a new Chef server set up on an Ubuntu 14.04 server.

The first step is downloading the required Chef server installer version. Visit the Chef server page at this link https://downloads.chef.io/chef-server/ to find and download the version you will use for your installation.

Next, you will want to transfer the downloaded installer package to your soon to be Chef server.

Once the installer is available on your server, you will want to install the package. The command to do this is:

sudo dpkg -i chef-server-core_12.2.0-1_amd64.deb

Once the Chef server package has been installed onto your server, the first thing to do is start the Chef server services. The command to do this is:

sudo chef-server-ctl reconfigure

The Chef server startup does a lot of work and as such you will see a lot of activity displayed as the setup progresses. When the startup command finally finishes successfully (a few minutes later; you might want to get a cup of coffee), the result will be a running Chef server.

Are we there yet? Well, we do have a Chef server up-and-running, but it is not much good to us yet. If you browse to your new Chef server, you will see something like this:

That is a good start, but there is still work to be done. Next up is creating your first Chef user on the Chef server with the user-create command. That user will be the initial Chef admin for the server. The format of the user-create command is:

sudo chef-server-ctl user-create user_name first_name last_name email password --filename FILE_NAME

Here is an example of what that command might look like:

sudo chef-server-ctl user-create earlwaud Earl Waud earl@sandiegoearl.com MyPassword99 --filename /home/earlwaud/earlwaud.pem

Here is what it looks like when you issue the command on your Chef server (along with the user-show command):

The filename parameter used in the user-create command provides the location at which to store the user's private key. The contents of this key file should be securely stored so that they can be provided to the user whose account has been created.

The next step is to create the first organization in your Chef server. The format of that command is as follows:

sudo chef-server-ctl org-create short_name "full_organization_name" --association_user user_name --filename ORGANIZATION-validator.pem

Here is the command I used to create the organization on my example Chef server:

sudo chef-server-ctl org-create sdearl "San Diego Earl" --association_user earlwaud --filename /home/earlwaud/sdearl-validator.pem

You can see that, in the example, the key file was saved at /home/earlwaud/sdearl-validator.pem.

If the filename parameter is not provided in the org-create command, the validator key will be displayed to stdout.

Either way, it is vital that the key information be captured and securely stored since it is not saved in the Chef server database and cannot be recovered if lost. The only option available when an organization's private key is completely lost is to reset the validation key.

Next, we need to install a package that will allow management of the Chef server via a web interface. The package is named opscode-manage. The command to install the package is:

sudo chef-server-ctl install opscode-manage

Now we need to restart the Chef server services by issuing the reconfigure command again:

sudo chef-server-ctl reconfigure

And finally, we need to start the opscode-manage services. This is done by issuing the command:

sudo opscode-manage-ctl reconfigure

If everything went as planned, you should now be able to browse to the new Chef server, and log in with your newly created admin user. Of course, there is still more work to do, such as integrating your new Chef server with your corporate domain to allow domain user access control. Still, this is a good start, right?

Knife is a command-line tool that supplies the interface between your chef-repo and your Chef server. It allows developers to upload their work on the workstation to the Chef server, from where it can be distributed to the nodes during their Chef client runs. Knife also allows you to obtain cookbooks from the supermarket, and it provides a mechanism to bootstrap nodes.

Some of the most commonly used community tools are installed as part of the ChefDK. Test Kitchen is installed, so you can test the results of your recipes and cookbook development on deployed test nodes. Berkshelf is installed to manage your cookbooks and their dependencies. Foodcritic is included in the ChefDK installation. Foodcritic will help you check your cookbooks for common errors and omissions.

Another valuable community tool installed with the ChefDK is Rubocop. You can use it to give you cookbooks some "style" and make sure that you're following the same conventions used by other developers in your organization. There is also ChefSpec, which is based on the ruby tool RSpec. ChefSpec will allow you to unit-test your cookbooks and recipes. We will have to exercise a lot of these community tools in later chapters, so please stay tuned.

You are going to download Chef Development Kit by opening your favorite browser and visiting the Chef.io ChefDK downloads page found at this URL: https://downloads.chef.io/chef-dk/.

To initiate the installer for the Chef Development Kit, double-click on the mounted ChefDK image icon.

The installer will now go about its business of installing the super fantastic Chef Development Kit, including the corresponding versions of community tools such as Test Kitchen, Food Critic, and Rubocup.

Finally, you will get the installation summary screen congratulating you on your successful installation and keen DevOps Kung-Fu.

The captain has turned off the seat-belt sign, so it is now OK to unmount the ChefDK image, and move about the installation.

sudo dpkg -i ~/Downloads/chefdk_0.12.0-1_amd64.deb

The ChefDK installer is a superhero for DevOps aficionados, saving them time and energy by installing the major necessities for a Chef workstation. Still, let's double-check the work done by the installer to make sure that we are really ready to go.

First, let's just try a simple command to see if things look right. Open a terminal window and issue the command chef --help. If the install was successful, you should see information about using Chef.

Next, let's use the new Chef tool to do a deeper validation of the install. Issue the following command in your terminal session:

chef verify

You should see the Chef tool validate each of the major packages installed during the ChefDK installation. Note that, depending on the state of your (OS X) workstation at the time you run this command, you may receive an error message regarding the need to install the command-line tools for gcc. It may look something like this:

If you get a message such as the preceding one, go ahead and install the command-line tools and then run the chef verify command again. This time, you should get a clean run of the command, and the output should be very similar to that shown in the preceding screenshot.

Now, at this point, things are looking pretty good, and you should be feeling fine. So, let's do one last validation check just to be 100% certain. Let's check the versions of some of the expected tools installed via the ChefDK installer.

Issue the following commands:

chef --version
ohai --version
foodcritic --version
rubocop --version
rspec --version

Your results should look something like this:

Hey wait… what happened to rspec? The Version 0.12 ChefDK installer does not seem to have successfully installed rspec, which is a requirement for the Chef-specific tool ChefSpec. That's not good. What went wrong? Everything seemed to have been installed based on the verify command we just used. What gives?

Well, rspec is installed, there is just one more step that we need to do. By the way, I think this is something that should be done by the ChefDK installer, but for whatever reason it's not, and it falls on our shoulders to handle it.

So what do we have to do? We need to set up the environment used in our terminal window for our Chef workstation development. How do we do that?

Fortunately, Chef has a command for that. It is the chef shell-init command. You can run this command interactively every time you open a new terminal window, or you can update your profile so that it is run automatically. I would recommend updating your profile, but let me show you both ways so you can make the call.

To make the interactive change, you would issue a command in the following format:

eval "$(chef shell-init SHELL_NAME)"

The actual call will be the following for the bash shell:

eval "$(chef shell-init bash)"

If you want to make the change more permanent, then you need to issue a command in the following format:

echo 'eval "$(chef shell-init SHELL_NAME)"' >> ~/.YOUR_SHELL_RC_FILE

If you are using Mac, the commands will be:

echo 'eval "$(chef shell-init bash)"' >> ~/.bash_profile
source ~/.bash_profile

If you are using Ubuntu, the commands will be:

echo 'eval "$(chef shell-init bash)"' >> ~/.bashrc
source ~/.bashrc

This echo command will add the eval command to your bash startup so that each run of your bash session will already have the desired environment ready to go for your Chef development work.

It should look something like this:

By the way, when you install the ChefDK onto a Windows system, the installation creates a Chef Development Kit icon on your desktop. When you execute from icon, a Chef-specific PowerShell session is launched. It will automatically run the chef shell-init command and set up the full environment needed for Chef. So there is no need to do this last step on a Windows workstation.

Well, there you have it: the successful installation of the ChefDK. Your workstation is ready to do some real Chef work now. You should feel pretty awesome!

Another benefit of the new Chef command-line tool is that it will create a standard chef repo for you. You no longer need to download or clone a starter repo from GitHub. You can use the chef generate command to create your new repo. Let's take a look at the chef generate repo command's help:

chef generate repo -help

Based on the description of the default values for the chef generate repo command, we can simply issue the following command on our workstation:

chef generate repo ~/chef-repo

That was easy! So what do we get? Well, let's have a look:

tree -a chef-repo
chef-repo
├── .chef-repo.txt
├── .git
│   ├── HEAD
│   ├── branches
│   ├── config
│   ├── description
│   ├── hooks
│   │   ├── applypatch-msg.sample
│   │   ├── commit-msg.sample
│   │   ├── post-update.sample
│   │   ├── pre-applypatch.sample
│   │   ├── pre-commit.sample
│   │   ├── pre-push.sample
│   │   ├── pre-rebase.sample
│   │   ├── prepare-commit-msg.sample
│   │   └── update.sample
│   ├── info
│   │   └── exclude
│   ├── objects
│   │   ├── info
│   │   └── pack
│   └── refs
│       ├── heads
│       └── tags
├── .gitignore
├── LICENSE
├── README.md
├── chefignore
├── cookbooks
│   ├── README.md
│   └── example
│       ├── README.md
│       ├── attributes
│       │   └── default.rb
│       ├── metadata.rb
│       └── recipes
│           └── default.rb
├── data_bags
│   ├── README.md
│   └── example
│       └── example_item.json
├── environments
│   ├── README.md
│   └── example.json
└── roles
    ├── README.md
    └── example.json

This list of files looks pretty good but, as you may have noted, an essential folder is missing: the .chef folder, which needs to contain the private key files and the knife.rb file, used to communicate with the Chef server. Let's take a look at creating or obtaining these files. One way is to use the starter kit. It is an OK way if this is a new Chef server. But, if it's not, be sure to read the following Starter Kit info; it could save your job!

You still need the .chef files for your new workstation to securely communicate with your Chef server. These files are your user private key file, the organization's validator private key file, and the knife.rb file. These files need to be placed in a .chef folder in your chef-repo.

When setting up a new organization in your Chef server, there is an easy way to get the files: with the repo starter kit.

Warning!!! Only download the starter kit if you are the Chef server admin and are the first (or only) person setting up your repo for a brand new organization. Currently, the act of downloading the starter kit will reset both your user private key and the organization validator key.

Downloading the starter kit on a mature organization will essentially break the Internet.

With this warning, if you are still going to use the starter kit, proceed at your own risk.

To get the starter kit, you will want to browse to your Chef server and log in. Once there, click on Administration and then select your Organization. Now click on Starter Kit and you will see the Download Starter Kit button. When you click the button, you will get the warning shown earlier. Click on the Proceed button to transfer a zip archive to your workstation, then unzip it into your chef-repo folder, and you're done.

Really, the only files you need from the starter kit are the .chef folder files:

The organization's key file can be provided by an administrator of the Chef server. The organization's validator key is generated when the organization is created. When the command-line tool is used to create the org, the location of the validator key file may be provided in the command. Otherwise, it will be displayed to stdout. The Chef server admin will have captured the validator key info and will be able to provide it to you now.

The user private key is much like the organization's private key in that it is created when the user is created and must be captured and securely stored at that time as it is not saved to the Chef server database. A user can reset their private key through the use of the Reset Key feature available when viewing the user profile on a Chef server. A user resetting their private key will have no negative impact on other users or the organization, so it is relatively safe to do.

Finally, you need the knife.rb file. This file is the easy one. It can be downloaded from the Chef server any time by logging in and browsing to the organization. From there, you can click on any of the Generate Knife Config links in the UI. This will create the knife.rb file and download it to your workstation. Another option is to use the knife configure command. This will prompt you through the process of creating your knife.rb file.

Put all three of these files into the .chef folder of your new chef-repo. Make sure that you save the two key files using the names in the knife.rb file (or conversely, update the contents of the knife.rb file to match the file names used to save your key files).

OS X doesn't come with a version of the tree command installed so, if you like that tool and use a Mac, I've found a clever way to have your cake and eat it too by creating your own tree script. This script was used to create the "tree" output shown in the preceding screenshot. This idea was shared by users Ahmed Masud and JakeGould on http://superuser.com. It goes like this:

When you are using an On Premise Chef server, you need to do one more thing to finish setting up your chef-repo. If the Chef server you are connecting to has a self-signed certificate, then you will need to add the server's certificate to your chef-repo to allow your workstation to trust your Chef server. An easy way to check if this applies to you is to run the ssl check command. It goes like this:

knife ssl check

If the Chef Server you are communicating with has a self-signed certificate, you will see a message like this:

As you can see in the error message generated by the ssl check command, the Chef server's certificate needs to be obtained and placed into a trusted_certs folder in your chef-repo/.chef folder. Again, you will need to contact your Chef server administrator to get a copy of the certificate file. Once you have it, just place it in the trusted certs folder; when you run your ssl check command again, you should get a message back that says something like:

knife ssl check
Connecting to host chefserver:443
Successfully verified certificates from 'chefserver'

With that successful knife command completed, you have verified that you have successfully set up your chef-repo and are ready to start creating your infrastructure as code.

Everyone has their favorite text editor. For a lot of people that editor is Sublime Text, but there are many other choices available. A lot of old schoolers still prefer to use a flavor of the vi editor. Many OS X users swear by Mac-only TextMate. And now there's a new kid on the block from the makers of GitHub: Atom 1.0. Sublime Text and Atom are both available for multiple platforms, so whatever OS is your poison, they have a version for you.

The project space feature is one of the many reasons I usually recommend Sublime Text, but this feature is also present in other editors such as the new Atom 1.0.

To keep things current, I will use the Atom 1.0 editor in the remainder of this book's examples.

Once you have decided on the editor you plan to use for your Chef workstation and have that editor installed and working, there is one more step. You will need to set the configuration to allow you to integrate your editor with the Chef command-line tools. The best way to do that is to edit your .chef/knife.rb file and add a line to set the editor you want to use with your knife commands. (You could also add an export EDITOR= command to your shell's .bashrc or .bash_profile file.)

Here are some sample knife.rb entries:

Sublime Text:

knife[:editor] = '/Applications/Sublime\ Text.app/Contents/SharedSupport/bin/subl -w'

Atom:

knife[:editor] = '/Applications/Atom.app/Contents/MacOS/Atom -w'

On Mac, the Atom editor has a built-in feature to create symbolic links in the /usr/local/bin folder. With the Atom editor running, open the Atom menu and select Install Shell Commands from the submenu. This will create two new links for you. If you do this before updating your knife.rb file, the knife[:editor] entry simply becomes this:

knife[:editor] = 'atom -w'

When you're using a Windows OS for your workstation, you will have to take greater care in formatting your knife[:editor] line to escape the slash characters in the path. A convenient way to do that is to use single quotes around the full value and use double quotes for the path. For Sublime Text, it might look something like this:

knife[:editor] = '"C:\Program Files\Sublime Text 2\sublime_text.exe" --wait'

You can test your editor setting by issuing a knife command that requires using the editor. For example, try the command:

knife client create editor-tester

If your editor integration is configured correctly, you will see your editor open up and show you some code. Save the code and exit the editor, and you should see the private key value print out. (This key is not important at this time because we are just testing the editor, and will immediately delete the client.)

If your editor opens, you are done. If you get an error that says Please set EDITOR environment variable, then your knife[:editor] value is incorrect. There is probably a typo in the path or some other error. Double-check the value and try again until you successfully launch the editor with your knife client create command.

Once you have validated the integration with your editor, you can delete the test client with the following command:

knife client delete editor-tester

Now you are ready to edit your infrastructure code! When you are creating amazing infrastructure code, you are going to want to make sure that it gets preserved and to ensure that happens you going to want to use a source control tool. Say hello to my little friend, Git.

Git, initially designed and developed by Linus Torvalds for Linux kernel development back in 2005, is now the most widely used version control system for developers everywhere. It's the number-one choice due to its speed, data integrity, and its distributed workflow model. Git is certainly the primary choice as the version control system to be used when developing infrastructure code with Chef.

I will be emphasizing the use of Git throughout the examples in the rest of the book. Git is included with Mac OS X, but it will need to be installed on Windows and Ubuntu workstations. I recommend you to use the GitHub desktop installer where available (Windows and OS X) and allow it to deploy the command-line tools as part of the installation. This installation of Git is easy and will be left to the reader to execute.

There are several virtualization choices you can use for your Chef testing. The overall goal is to have a system that you can use to deploy a temporary Chef node to test your infrastructure code. One of the best ways to accomplish this is to install a local virtualization tool on your workstation. Among the choices for a local hypervisor are:

One of the best choices for this kind of Chef testing is VirtualBox. The reasons include its functionality, its ease of use, and its free price tag. Either of the VMware solutions will cost you not only for the hypervisor layer, but also for the integration plugins we'll use with Vagrant to manage our test nodes.

Visit the VirtualBox downloads page at https://www.virtualbox.org/wiki/Downloads to download the installer. Once downloaded, it is an easy matter to install it, so I'll leave that to the reader to take care of.

Last, but certainly not least, is Vagrant. Vagrant is another item I consider to be one of the key pieces of the Chef testing puzzle. It is the "glue" between Chef and the hypervisor that deploys the test nodes during testing. You will want to download and install Vagrant on your workstation. You can find the downloaders at https://www.vagrantup.com/downloads.html. Vagrant is also an easy install, so I'll leave that to the reader to handle.

I'll go into using Vagrant with VitualBox (and Docker) in some detail in a later chapter.

One use case for chef-apply is to learn what a recipe does. You can use chef-apply to run a recipe in a "test only" mode by using the -W or --why-run parameter. This will converge the recipe, showing you everything that would happen with the recipe without making actual changes to your system.

You can also execute some one-liner scripts. You can easily use chef-apply in a scripting-like manner to install software. For example, if you need to use Git on your system and you find it's not installed. Just issue the command:

sudo chef-apply -e "package 'git'"

This will check to see if Git is currently installed. If it is, then the command will just inform you that that is the case and exit. If Git is not installed, then it will go about using the correct installer (aka Provider) to obtain and install the package.

Testing this on my Ubuntu workstation, you can see that Git was not installed initially. Then, by using the chef-apply command, the git package is installed. Then to confirm that nothing happens if you try to use chef-apply to install a package that is already installed, I ran the same chef-apply again to show what happens:

How about creating a "workstation" recipe to set up all the "missing" items from a new ChefDK workstations setup? You create the recipe that defines the state of having your editor installed and integrated with Chef, defines the state of having git installed, and defines the state of having your custom OS X "tree" script created in /usr/local/bin. You create this recipe once, and you (and anyone in your organization) can quickly complete their workstation setup by using chef-apply with your workstation recipe.