Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases now! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Conferences
Free Learning
Arrow right icon
Chef Infrastructure Automation Cookbook Second Edition
Chef Infrastructure Automation Cookbook Second Edition

Chef Infrastructure Automation Cookbook Second Edition: Over 80 recipes to automate your cloud and server infrastructure with Chef and its associated toolset

eBook
€20.98 €29.99
Paperback
€36.99
Subscription
Free Trial
Renews at €18.99p/m

What do you get with a Packt Subscription?

Free for first 7 days. €18.99 p/m after that. Cancel any time!
Product feature icon Unlimited ad-free access to the largest independent learning library in tech. Access this title and thousands more!
Product feature icon 50+ new titles added per month, including many first-to-market concepts and exclusive early access to books as they are being written.
Product feature icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Product feature icon Thousands of reference materials covering every tech concept you need to stay up to date.
Subscribe now
View plans & pricing
Table of content icon View table of contents Preview book icon Preview Book

Chef Infrastructure Automation Cookbook Second Edition

Chapter 1. Chef Infrastructure

"What made Manhattan Manhattan was the underground infrastructure, that engineering marvel."

Andrew Cuomo

A well-engineered infrastructure builds the basis for successful companies. In this chapter, we will see how to set up the infrastructure around Chef as the basis of your infrastructure as code. We'll cover the following recipes in this chapter:

  • Using version control
  • Installing the Chef development kit on your workstation
  • Using the hosted Chef platform
  • Managing virtual machines with Vagrant
  • Creating and using cookbooks
  • Inspecting files on your Chef server with knife
  • Defining cookbook dependencies
  • Managing cookbook dependencies with Berkshelf
  • Downloading and integrating cookbooks as vendor branches into your Git repository
  • Using custom knife plugins
  • Deleting a node from the Chef server
  • Developing recipes with local mode
  • Using roles
  • Using environments
  • Freezing cookbooks
  • Running Chef client as a daemon
  • Using chef-shell

Introduction

This chapter will cover the basics of Chef, including common terminology, workflow practices, and various tools that work in accordance with Chef. We will explore version control using Git, walk through working with community cookbooks, and running those cookbooks on your own servers, so that you can configure them in the way you need them.

First, let's talk about some important terms used in the Chef universe.

A cookbook is a collection of all the components needed to change something on a server, such as installing MySQL, the most important one being recipes, which tell Chef which resources you want to configure on your host.

You need to deploy cookbooks to the nodes that you want to change. Chef offers multiple ways for this task. Most probably, you'll use a central Chef server. You can either run your own server or sign up for hosted Chef.

The Chef server is the central registry, where each node needs to be registered. The Chef server distributes the cookbooks you uploaded to it, to your nodes.

Knife is Chef's command-line tool to interact with the Chef server. You run it on your local workstation and use it to upload cookbooks and manage other aspects of Chef.

On your nodes, you need to install Chef client—the part that retrieves the cookbooks from the Chef server and executes them on the node.

In this chapter, we'll see the basic infrastructure components of your Chef setup at work and learn how to use the basic tools. Let's get started by taking a look at how to use Git as a version control system for your cookbooks.

Using version control

Do you manually back up every file before you change it? And do you invent creative file name extensions such as _me and _you when you try to collaborate a file? If you answer yes to any of these, it's time to rethink your processes.

A version control system (VCS) helps you stay sane when dealing with important files and collaborating with them.

Using version control is a fundamental part of any infrastructure automation. There are multiple solutions (some free, some paid) to manage source version control, including Git, SVN, Mercurial, and Perforce. Due to its popularity among the Chef community, we will be using Git. However, you could easily use any other version control system with Chef.

Note

Don't even think about building your infrastructure as code without using a version control system to manage it!

Getting ready

You'll need Git installed on your local workstation. Either use your operating system's package manager (such as Apt on Ubuntu or Homebrew on OS X), or simply download the installer from www.git-scm.org.

Git is a distributed version control system. This means that you don't necessarily need a central host to store your repositories. However, in practice, using GitHub as your central repository has proven to be very helpful. In this book, I'll assume that you're using GitHub. Therefore, you need to go to www.github.com and create an (free) account to follow the instructions given in this book. Make sure that you upload your Secure Shell (SSH) key by following the instructions at https://help.github.com/articles/generating-ssh-keys, so that you're able to use the SSH protocol to interact with your GitHub account.

As soon as you have created your GitHub account, you should create your repository by visiting https://github.com while you're still logged in and using chef-repo as the repository name.

Make sure you have wget installed on your local workstation, in order to be able to download the required files from public servers.

How to do it...

Before you can write any cookbooks, you need to set up your initial Git repository on your development box. Chef Software, Inc. provides an empty Chef repository to get you started. Let's see how you can set up your own Chef repository with Git, using Chef's skeleton.

  1. Download Chef's skeleton repository as a tarball:
    mma@laptop
     $ wget http://github.com/chef/chef-repo/tarball/master
    ...TRUNCATED OUTPUT...
    2014-11-30 22:00:43 (1.30 MB/s) - 'master' saved [9309/9309]
    
  2. Extract the downloaded tarball:
    mma@laptop $ tar xzvf master
    
  3. Rename the directory:
    mma@laptop:~ $ mv opscode-chef-repo-* chef-repo
    
  4. Change to your newly created Chef repository:
    mma@laptop:~ $ cd chef-repo/
    
  5. Initialize a fresh Git repository:
    mma@laptop:~/chef-repo $ git init .
    Initialized empty Git repository in /Users/mma/work/chef-repo/.git/
    
  6. Connect your local repository to your remote repository on github.com. Make sure to replace mmarschall with your own GitHub username:
    mma@laptop:~/chef-repo $ git remote add origin git@github.com:mmarschall/chef-repo.git
    
  7. Configure Git with your user name and e-mail address:
    mma@laptop:~/chef-repo $ git config --global user.email "you@example.com"
    mma@laptop:~/chef-repo $ git config --global user.name "Your Name"
    
  8. Add and commit Chef's default directory structure:
    mma@laptop:~/chef-repo $ git add .
    mma@laptop:~/chef-repo $ git commit -m "initial commit"
    
    [master (root-commit) 6148b20] initial commit
     11 files changed, 545 insertions(+), 0 deletions(-)
     create mode 100644 .gitignore
    ...TRUNCATED OUTPUT...
    create mode 100644 roles/README.md
  9. Push your initialized repository to GitHub. This makes it available to all your co-workers to collaborate on:
    mma@laptop:~/chef-repo $ git push -u origin master
    
    ...TRUNCATED OUTPUT...
    To git@github.com:mmarschall/chef-repo.git
     * [new branch]      master -> master

How it works...

You have downloaded a tarball containing Chef's skeleton repository. Then, you initialized chef-repo and connected it to your own repository on GitHub.

After that, you added all the files from the tarball to your repository and committed them. This makes Git track your files and the changes you make later.

Finally, you pushed your repository to GitHub, so that your co-workers can use your code too.

There's more...

Let's assume you're working on the same chef-repo repository, together with your co-workers. They cloned your repository, added a new cookbook called other_cookbook, committed their changes locally, and pushed their changes back to GitHub. Now, it's time for you to get the new cookbook downloaded on to your own laptop.

Pull your co-workers' changes from GitHub. This will merge their changes into your local copy of the repository. Use the pull subcommand:

mma@laptop:~/chef-repo $ git pull --rebase
From github.com:mmarschall/chef-repo
 * branch            master     -> FETCH_HEAD
...TRUNCATED OUTPUT...
create mode 100644 cookbooks/other_cookbook/recipes/default.rb

In case of any conflicting changes, Git will help you merge and resolve them.

See also

Installing the Chef development kit on your workstation

If you want to use Chef, you'll need to install the Chef development kit (DK) on your local workstation first. You'll have to develop your configurations locally and use Chef to distribute them to your Chef server.

Chef provides a fully packaged version, which does not have any external prerequisites. This fully packaged Chef is called the omnibus installer. We'll see how to use it in this section.

How to do it...

Let's see how to install the Chef DK on your local workstation using Chef's omnibus installer:

  1. Download the Chef DK for your specific workstation platform from https://downloads.chef.io/chef-dk/ and run the installer.
  2. Verify that Chef installed all the required components:
    mma@laptop:~ $ chef verify
    
    ...TRUNCATED OUTPUT...
    Verification of component 'rubocop' succeeded.
    Verification of component 'kitchen-vagrant' succeeded.
    Verification of component 'chefspec' succeeded.
    Verification of component 'berkshelf' succeeded.
    Verification of component 'fauxhai' succeeded.
    Verification of component 'test-kitchen' succeeded.
    Verification of component 'package installation' succeeded.
    Verification of component 'chef-dk' succeeded.
    Verification of component 'knife-spork' succeeded.
    Verification of component 'chef-client' succeeded.
  3. Add the newly installed Ruby to your path:
    mma@laptop:~ $ echo 'export PATH="/opt/chefdk/embedded/bin:$PATH"' >> ~/.bash_profile && source ~/.bash_profile
    

    Note

    You may not want to use (and don't have to use) ChefDK's Ruby, especially if you are a Rails Developer.

    If you're happily using your Ruby rvm, or rbenv environment, you can continue to do so. Just ensure that the ChefDK-provided applications appear first in your PATH, before any gem-installed versions, and you're good to go.

How it works...

The omnibus installer will download Ruby and all required Ruby gems into /opt/chefdk.

See also

Using the hosted Chef platform

If you want to get started with Chef right away (without the need to install your own Chef server) or want a third party to give you a Service Level Agreement (SLA) for your Chef server, you can sign up for hosted Chef by Chef Software, Inc. Chef Software, Inc. operates Chef as a cloud service. It's quick to set up and gives you full control, using users and groups to control the access permissions to your Chef setup. We'll configure knife, Chef's command-line tool to interact with hosted Chef, so that you can start managing your nodes.

Getting ready

Before being able to use hosted Chef, you need to sign up for the service. There is a free account for up to five nodes.

Visit http://manage.chef.io/signup and register for a free trial or a free account.

I registered as the user webops with an organization short name of awo.

After registering your account, it is time now to prepare your organization to be used with your chef-repo repository.

How to do it...

Carry out the following steps in order to interact with the hosted Chef:

  1. Create the configuration directory for your Chef client on your local workstation:
    mma@laptop:~/chef-repo $ mkdir .chef
    
  2. Navigate to http://manage.chef.io/organizations. After logging in, you can start downloading your validation keys and configuration file.
  3. Select your organization to be able to see its contents using the web UI.
    How to do it...

    Regenerate the validation key for your organization and save it as <your-organization-short-name>-validator.pem in the chef directory inside your chef-repo repository.

    How to do it...
  4. Generate the knife config and put the downloaded knife.rb into the .chef directory inside your chef-repo directory, as well. Make sure you have downloaded your user's private key from https://www.chef.io/account/password and replace webops with the username you chose for hosted Chef, and awo with the short name you chose for your organization:
    current_dir = File.dirname(__FILE__)
    log_level                :info
    log_location             STDOUT
    node_name                "webops"
    client_key               "#{current_dir}/webops.pem"
    validation_client_name   "awo-validator"
    validation_key           "#{current_dir}/awo-validator.pem"
    chef_server_url          "https://api.chef.io/organizations/awo"
    cache_type               'BasicFile'
    cache_options( :path => "#{ENV['HOME']}/.chef/checksums" )
    cookbook_path            ["#{current_dir}/../cookbooks"]

    Tip

    Take a look at the following code:

    .chef/*.pem
    .chef/encrypted_data_bag_secret

    You should add the preceding code to your .gitingore file inside chef-repo to avoid your credentials from ending up in your Git repository.

  5. Use knife to verify that you can connect to your hosted Chef organization. It should only have your validator client, so far. Instead of awo, you'll see your organization's short name:
    mma@laptop:~/chef-repo $ knife client list
    awo-validator
    

How it works...

Hosted Chef uses two private keys (called validators):

  • one for the organization
  • one for every user.

You need to tell knife where it can find these two keys in your knife.rb file.

The following two lines of code in your knife.rb file tell the knife about which organization to use and where to find its private key. The validation_key is used to allow new clients to authenticate the Chef server before getting their own Client key:

validation_client_name   "awo-validator"
validation_key           "#{current_dir}/awo-validator.pem"

The following line of code in your knife.rb file tells the knife where to find your users' private key. It is used by your local workstation to authenticate the Chef server:

client_key               "#{current_dir}/webops.pem"

Also, the following line of code in your knife.rb file tells knife that you are using hosted Chef. You will find your organization name as the last part of the URL:

chef_server_url          "https://api.chef.io/organizations/awo"

Using the knife.rb file and your two validators knife, you can now connect to your organization hosted by Chef Software, Inc.

You do not need your own self-hosted Chef server, nor do you need to use Chef client local mode in this setup.

There's more...

This setup is good for you if you do not want to worry about running, scaling, and updating your own Chef server and if you're happy with saving all your configuration data in the Cloud (under the control of Chef Software, Inc.).

Note

If you need to have all your configuration data within your own network boundaries, you can install Chef server on premises by choosing "ON PREMISES CHEF" at https://www.chef.io/chef/choose-your- version/ or install the Open Source version of Chef server directly from GitHub at https://github.com/chef/chef.

See also

Managing virtual machines with Vagrant

Developing Chef cookbooks requires you to run your work in progress cookbooks multiple times on your nodes. To make sure they work, you need a clean, initial state of your nodes every time you run them. You can achieve this by using a virtual machine (VM). However, manually setting up and destroying VMs is tedious and breaks your development flow.

Vagrant is a command-line tool, which provides you with a configurable, reproducible, and portable development environment by enabling you to manage VMs. It lets you define and use preconfigured disk images to create new VMs from. Also, you can configure Vagrant to use provisioners such as Shell scripts, Puppet, or Chef to bring your VM into the desired state.

In this recipe, we will see how to use Vagrant to manage VMs using VirtualBox and Chef client as the provisioner.

Getting ready

Download and install VirtualBox at https://www.virtualbox.org/wiki/Downloads.

Download and install Vagrant at https://www.vagrantup.com/downloads.html.

Install the Omnibus Vagrant plugin to enable Vagrant to install the Chef client on your VM by running the following command:

mma@laptop:~/chef-repo $ vagrant plugin install vagrant-omnibus
Installing the 'vagrant-omnibus' plugin. This can take a few minutes...
Installed the plugin 'vagrant-omnibus (1.4.1)'!

How to do it...

Let's create and boot a virtual node by using Vagrant:

  1. Visit https://github.com/opscode/bento and choose a Vagrant box to base your VMs on. We'll use opscode-ubuntu-14.10 in this example.
  2. The URL of the opscode-ubuntu-14.10 box is https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.10_chef-provisionerless.box.
  3. Edit your new Vagrantfile. Make sure that you replace <YOUR-ORG> with the name of your organization on the Chef server. Use the name and URL of the box file you noted down in the first step as config.vm.box and config.vm.box_url:
    mma@laptop:~/chef-repo $ subl Vagrantfile
    
    Vagrant.configure("2") do |config|
      config.vm.box = "opscode-ubuntu-14.10"
      config.vm.box_url = "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.10_chef-provisionerless.box"
      config.omnibus.chef_version = :latest
    
      config.vm.provision :chef_client do |chef|
        chef.provisioning_path = "/etc/chef"
        chef.chef_server_url = "https://api.chef.io/organizations/<YOUR_ORG>"
        chef.validation_key_path = ".chef/<YOUR_ORG>-validator.pem"
        chef.validation_client_name = "<YOUR_ORG>-validator"
        chef.node_name = "server"
      end
    end
  4. Create your virtual node using Vagrant:
    mma@laptop:~/chef-repo $ vagrant up
    
    Bringing machine 'server' up with 'virtualbox' provider...
    ...TRUNCATED OUTPUT...
    ==> default: Importing base box 'opscode-ubuntu-14.10'...
    ...TRUNCATED OUTPUT...
    ==> default: Installing Chef 11.16.4 Omnibus package...
    ...TRUNCATED OUTPUT...
    ==> default: Running provisioner: chef_client...
    ==> default: Creating folder to hold client key...
    ==> default: Uploading chef client validation key...
    Generating chef JSON and uploading...
    ==> default: Running chef-client...
    ==> default: [2014-12-01T22:00:54+00:00] INFO: *** Chef 11.16.4 ***
    ...TRUNCATED OUTPUT...
  5. Log in to your virtual node using SSH:
    mma@laptop:~/chef-repo $ vagrant ssh
    
    Welcome to Ubuntu 14.10 (GNU/Linux 3.16.0-23-generic x86_64)
    
     * Documentation:  https://help.ubuntu.com/
    Last login: Mon Oct 27 02:22:37 2014 from 10.0.2.2
    
    vagrant@server:~$
  6. Log out of your virtual node:
    vagrant@server:~$ exit
    mma@laptop:~/chef-repo $
    
  7. Validate that the Chef server knows your new virtual machine as a client called server:
    mma@laptop:~/chef-repo $ knife client list
    
    awo-validator
    server
  8. Go to https://manage.chef.io/organizations/<YOUR ORGANIZATION>/nodes and validate that your new virtual machine shows up as a registered node:
    How to do it...

How it works...

The Vagrantfile is written in a Ruby Domain Specific Language (DSL) to configure the Vagrant virtual machines. We want to boot a simple Ubuntu VM. Let's go through the Vagrantfile step by step.

First, we create a config object. Vagrant will use this config object to configure the VM:

Vagrant.configure("2") do |config|
end

Inside the config block, we tell Vagrant which VM image to use, in order to boot the node:

config.vm.box = "opscode-ubuntu-14.10"
config.vm.box_url = "https://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-14.10_chef-provisionerless.box"

We want to boot our VM using a so-called Bento Box, provided by Chef. We use Ubuntu Version 14.10 here.

Note

If you have never used the box before, Vagrant will download the image file (a few hundred megabytes) when you run vagrant up for the first time.

As we want our VM to have the Chef client installed, we tell the omnibus vagrant plugin to use the latest version of Chef client:

  config.omnibus.chef_version = :latest

After selecting the VM image to boot, we configure how to provision the box by using Chef. The Chef configuration happens in a nested Ruby block:

  config.vm.provision :chef_client do |chef|
  ...
  end

Inside this Chef block, we need to instruct Vagrant on how to hook up our virtual node to the Chef server. First, we need to tell Vagrant where to store all the Chef stuff on your node:

    chef.provisioning_path = "/etc/chef"

Vagrant needs to know the API endpoint of your Chef server. If you use hosted Chef, it is https://api.chef.io/organizations/<YOUR_ORG>. You need to replace <YOUR_ORG> with the name of the organization that you created in your account on hosted Chef. If you are using your own Chef server, change the URL accordingly:

    chef.chef_server_url = "https://api.chef.io/organizations/<YOUR_ORG>"

While creating your organization on hosted Chef, you must download your private key. Tell Vagrant where to find this file:

    chef.validation_key_path = ".chef/<YOUR_ORG>—validator.pem"

Also, you need to tell Vagrant as to which client it should validate itself against the Chef server:

    chef.validation_client_name = "<YOUR_ORG>-validator"

Finally, you should tell Vagrant how to name your node:

    chef.node_name = "server"

After configuring your Vagrantfile, all you need to do is run the basic Vagrant commands such as vagrant up, vagrant provision, and vagrant ssh. To stop your VM, just run the vagrant halt command.

There's more...

If you want to start from scratch again, you will have to destroy your VM and delete both the client and the node from your Chef server by running the following command:

mma@laptop:~/chef-repo $ vagrant destroy
mma@laptop:~/chef-repo $ knife node delete server -y && knife client delete server -y

Alternatively, you may use the Vagrant Butcher plugin found at https://github.com/cassianoleal/vagrant-butcher.

Tip

Don't blindly trust Vagrant boxes downloaded from the Web; you never know what they contain.

See also

Creating and using cookbooks

Cookbooks are an essential part of Chef. You can easily create them using the Chef executable installed by the Chef DK. In this recipe (and many of the following recipes), I will assume that you're using a Chef server to manage your infrastructure. You can either set up your own cookbook or use the hosted Chef as described previously. You'll use the command-line tool knife to interact with the Chef server.

In this recipe, we'll create and apply a simple cookbook using the Chef and knife command-line tools.

Getting ready

Make sure you have Chef DK installed and a node available for testing. Check out the installation instructions at http://learn.chef.io if you need help here.

Edit your knife.rb file (usually found in the hidden .chef directory) and add the following three lines to it, filling in your own values:

cookbook_copyright "your company"
cookbook_license "apachev2"
cookbook_email "your email address"

Note

The Apache 2 license is the most commonly found in cookbooks, but you're free to choose whichever suits your needs. If you put none as cookbook_license, knife will put "All rights reserved" into your recipe's metadata file.

Chef will use the preceding values as default whenever you create a new cookbook.

We assume that you have a node called server registered with your Chef server, as described in the Managing virtual machines with Vagrant section in this chapter.

How to do it...

Carry out the following steps to create and use cookbooks:

  1. Create a cookbook named my_cookbook by running the following command:
    mma@laptop:~/chef-repo $ chef generate cookbook cookbooks/my_cookbook
    
    Compiling Cookbooks...
    Recipe: code_generator::cookbook
    
    
    ...TRUNCATED OUTPUT...

    Before ChefDK was introduced, the only way to generate cookbooks was to use knife cookbook create my_cookbook

  2. Upload your new cookbook on the Chef server:
    mma@laptop:~/chef-repo $ knife cookbook upload my_cookbook
    
    Uploading my_cookbook    [0.1.0]
    Uploaded 1 cookbook.
  3. Add the cookbook to your node's run list. In this example, the name of the node is server:
    mma@laptop:~/chef-repo $ knife node run_list add server 'recipe[my_cookbook]'
    
    server:
      run_list: recipe[my_cookbook]
  4. Run the Chef client on your node:
    user@server:~$ sudo chef-client
    

    Tip

    If you're using a Vagrant VM as your server, you need to make sure to run vagrant up and vagrant ssh in order to be able to execute the Chef client on the node.

How it works...

The chef executable helps you to manage your local Chef development environment. We used it here to generate the cookbook.

Knife is the command-line interface for the Chef server. It uses the RESTful API exposed by the Chef server to do its work and helps you to interact with the Chef server.

The knife command supports a host of commands structured as follows:

knife <subject> <command>

The <subject> used in this section is either cookbook or node. The commands we use are upload for the cookbook, and run_list add for the node.

See also

  • Learn how to set up your Chef server in the Using the hosted Chef platform recipe in this chapter

Inspecting files on your Chef server with knife

Sometimes, you may want to peek into the files stored on your Chef server. You might not be sure about an implementation detail of that specific cookbook version, which is currently installed on your Chef server, and need to look it up. Knife can help you out by letting you show various aspects of the files stored on your Chef server.

Getting ready

Make sure that you have the iptables cookbook installed locally and uploaded on your Chef server.

  1. Install the iptables community cookbook by executing the following command and code lines:
    mma@laptop:~/chef-repo $ knife cookbook site install iptables
    
    Installing iptables to /Users/mma/work/chef-repo/cookbooks
    ...TRUNCATED OUTPUT...

    Tip

    Take a look at the following error:

    ERROR: IOError: Cannot open or read ../chef-repo/cookbooks/iptables/metadata.rb!

    If you get the preceding error, your cookbook only has a metadata.json file. Make sure that you delete it and create a valid metadata.rb, file instead.

  2. Upload the iptables cookbook on your Chef server by executing the following given command and code lines:
    mma@laptop:~/chef-repo $ knife cookbook upload iptables
    
    Uploading iptables       [0.14.0]
    Uploaded 1 cookbook.

How to do it...

Let's find out how knife can help you to look into a cookbook stored in your Chef server:

  1. First, you want to find out the current version of the cookbook you're interested in. In our case, we're interested in the iptables cookbook:
    mma@laptop:~/work/chef_helpster $ knife cookbook show iptables
    iptables   0.14.0
    
  2. Then, you can look up the definitions of the iptables cookbook, using the version number that you found out in the previous step:
    mma@laptop:~/chef-repo $ knife cookbook show iptables 0.14.0 definitions
    
      checksum:     45c0b77ff10d7177627694827ce47340
      name:         iptables_rule.rb
      path:         definitions/iptables_rule.rb
      specificity:  default
      url:          https://s3.amazonaws.com/opscode-platform...
  3. Now, you can even show the contents of the iptables_rule.rb definition file, as stored on the server:
    mma@laptop:~/chef-repo $ knife cookbook show iptables 0.14.0 definitions iptables_rule.rb
    
    #
    # Cookbook Name:: iptables
    # Definition:: iptables_rule
    #
    #
    define :iptables_rule, :enable => true, :source => nil, :variables => {} do
    ...TRUNCATED OUTPUT...
    end

How it works...

The knife cookbook show subcommand helps you understand what exactly is stored on the Chef server. It lets you drill down into specific sections of your cookbooks and see the exact content of the files stored in your Chef server.

There's more...

Since Chef 11, you can pass patterns to the knife show command to tell it what exactly you want to see. Showing the contents of the iptables_rule definition can be done as follows, in addition to the way we described previously:

mma@laptop:~/work/chef_helpster $ knife show cookbooks/iptables/definitions/*
cookbooks/iptables/definitions/iptables_rule.rb:
#
# Cookbook Name:: iptables
# Definition:: iptables_rule
#
#
define :iptables_rule, :enable => true, :source => nil, :variables => {} do
...TRUNCATED OUTPUT...
end

See also

Defining cookbook dependencies

Quite often, you might want to use features of other cookbooks in your own cookbooks. For example, if you want to make sure that all packages required for compiling software written in C are installed, you might want to include the build-essential cookbook, which does just that. Chef server needs to know about such dependencies in your cookbooks. You declare them in a cookbook's metadata.

Getting ready

Make sure you have a cookbook named my_cookbook, and the run_list command of your node includes my_cookbook, as described in the Creating and using cookbooks recipe in this chapter.

How to do it...

Edit the metadata of your cookbook in the file cookbooks/my_cookbook/metadata.rb to add a dependency to the build-essential cookbook:

mma@laptop:~/chef-repo $ subl cookbooks/my_cookbook/metadata.rb
...
depends 'build-essential'
depends 'apache2', '>= 1.0.4'

How it works...

If you want to use a feature of another cookbook inside your cookbook, you will need to include the other cookbook in your recipe using the include_recipe directive:

include_recipe 'build-essential'

To tell the Chef server that your cookbook requires the build-essential cookbook, you need to declare that dependency in the metadata.rb file. If you uploaded all the dependencies to your Chef server either using berks install and berks upload or knife cookbook upload ..., the Chef server will then send all the required cookbooks to the node.

Tip

The first depends call tells the Chef server that your cookbook depends on the latest version of the build-essential cookbook.

The second depends call tells the Chef server that your cookbook depends on a version of the apache2 cookbook, which is greater or equal to the version 1.0.4. You may use any of these version constraints with your depends calls:

  • < (less than)
  • <= (less than or equal to)
  • = (equal to)
  • >= (greater than or equal to)
  • ~> (approximately greater than)
  • > (greater than)

There's more...

If you include another recipe inside your recipe, without declaring the cookbook dependency in your metadata.rb file, Foodcritic will warn you:

mma@laptop:~/chef-repo $ foodcritic cookbooks/my_cookbook
FC007: Ensure recipe dependencies are reflected in cookbook metadata: cookbooks/my_cookbook/recipes/default.rb:9

Tip

Foodcritic will just return an empty line, if it doesn't find any issues.

Additionally, you can declare conflicting cookbooks through the conflicts call:

conflicts "nginx"

Of course, you can use version constraints exactly the way you did with depends.

See also

  • Read more on how you can find out what is uploaded on your Chef server in the Inspecting files on your Chef server with knife recipe in this chapter
  • Find out how to use foodcritic in the Flagging problems in your Chef cookbooks recipe in Chapter 2, Evaluating and Troubleshooting Cookbooks and Chef Runs

Managing cookbook dependencies with Berkshelf

It's a pain to manually ensure that you have installed all the cookbooks that another cookbook depends on. You have to download each and every one of them manually only to find out that with each downloaded cookbook, you inherit another set of dependent cookbooks.

And even if you use knife cookbook site install, which installs all the dependencies locally for you, your cookbook directory and your repository get cluttered with all those cookbooks. Usually, you don't really care about all those cookbooks and don't want to see or manage them.

This is where Berkshelf comes into play. It works like Bundler for Ruby gems, managing cookbook dependencies for you. Berkshelf downloads all the dependencies you defined recursively and helps you to upload all cookbooks to your Chef server.

Instead of polluting your Chef repository, it stores all the cookbooks in a central location. You just commit your Berkshelf dependency file (called Berksfile) to your repository, and every colleague or build server can download and install all those dependent cookbooks based on it.

Let's see how to use Berkshelf to manage the dependencies of your cookbook.

Getting ready

Make sure you have a cookbook named my_cookbook and the run_list of your node includes my_cookbook, as described in the Creating and using cookbooks recipe.

How to do it...

Berkshelf helps you to keep those utility cookbooks out of your Chef repository. This makes it much easier to maintain the cookbooks, which really matter.

Let's see how to write a cookbook by running a bunch of utility recipes and manage the required cookbooks with Berkshelf:

  1. Edit your cookbook's metadata:
    mma@laptop:~/chef-repo $ subl cookbooks/my_cookbook/metadata.rb
    
    ...
    depends "chef-client"
    depends "apt"
    depends "ntp"
  2. Edit your cookbook's default recipe:
    mma@laptop:~/chef-repo $ subl cookbooks/my_cookbook/recipes/default.rb
    
    ...
    include_recipe "chef-client"
    include_recipe "apt"
    include_recipe "ntp"
  3. Run Berkshelf to install all the required cookbooks:
    mma@laptop:~/chef-repo $ cd cookbooks/my_cookbook
    mma@laptop:~/chef-repo/cookbooks/my_cookbook $ berks install
    
    Resolving cookbook dependencies...
    Fetching 'my_cookbook' from source at .
    Fetching cookbook index from https://supermarket.chef.io...
    Installing apt (2.6.0) from https://supermarket.chef.io ([opscode] https://supermarket.chef.io/api/v1)
    ...TRUNCATED OUTPUT...
  4. Upload all the cookbooks on the Chef server:
    mma@laptop:~/chef-repo/cookbooks/my_cookbook $ berks upload
    
    Using my_cookbook (0.1.0)
    ...TRUNCATED OUTPUT...
    Uploading windows (1.34.8) to: 'https://api.chef.io:443/organizations/awo

How it works...

Berkshelf comes with the Chef DK.

We create our cookbook and tell it to use a few basic cookbooks.

Instead of making us manually install all the cookbooks using knife cookbook site install, chef generate creates a Berksfile, besides the metadata.rb file.

The Berksfile is pretty simple. It tells Berkshelf to use the Chef supermarket as the default source for all cookbooks:

source "https://supermarket.chef.io"

It tells Berkshelf to read the metadata.rb file to find all the required cookbooks. This is the simplest way when working inside a single cookbook. Please see the following There's more… section to find an example of a more advanced usage of the Berksfile.

After telling Berkshelf where to find all the required cookbook names, we use it to install all those cookbooks:

berks install

Berkshelf stores cookbooks in ~/.berkshelf/cookbooks, by default. This keeps your Chef repository clutter-free. Instead of having to manage all the required cookbooks inside your own Chef repository, Berkshelf takes care of them. You simply need to check in Berksfile with your recipe, and everyone using your recipe can download all the required cookbooks by using Berkshelf.

To make sure that there's no mix-up with different cookbook versions when sharing your cookbook, Berkshelf creates a file called Berksfile.lock alongside Berksfile. Here, you'll find the exact versions of all the cookbooks that Berkshelf installed:

DEPENDENCIES
  my_cookbook
    path: .
    metadata: true

GRAPH
  apt (2.6.0)
  chef-client (3.9.0)
    cron (>= 1.2.0)
    logrotate (>= 1.2.0)
    windows (~> 1.11)
  chef_handler (1.1.6)
  cron (1.6.1)
  logrotate (1.7.0)
  my_cookbook (0.1.0)
    apt (>= 0.0.0)
    chef-client (>= 0.0.0)
    ntp (>= 0.0.0)
  ntp (1.6.8)
  windows (1.34.8)
    chef_handler (>= 0.0.0)

Berkshelf will only use the exact versions specified in the Berksfile.lock file, if it finds this file.

Finally, we use Berkshelf to upload all the required cookbooks on the Chef server:

berks upload

There's more...

Berkshelf integrates tightly with Vagrant via the vagrant-berkshelf plugin. You can set up Berkshelf and Vagrant in such a way that Berkshelf installs and uploads all the required cookbooks on your Chef server whenever you execute vagrant up or vagrant provision. You'll save all the work of running berks install and berks upload manually before creating your node with Vagrant.

Let's see how you can integrate Berkshelf and Vagrant.

First, you need to install the Berkshelf plugin for Vagrant:

mma@mma-mbp:~/work/chef-repo (master)$ vagrant plugin install vagrant-berkshelf
Installing the 'vagrant-berkshelf' plugin. This can take a few minutes...
Installed the plugin 'vagrant-berkshelf (4.0.1)'!

Then, you need to tell Vagrant that you want to use the plugin. You do this by enabling the plugin in Vagrantfile:

mma@mma-mbp:~/work/chef-repo (master)$ subl Vagrantfile
config.berkshelf.enabled = true

Then, you need a Berksfile in the root directory of your Chef repository to tell Berkshelf which cookbooks to install on each Vagrant run:

source 'https://supermarket.chef.io'

cookbook 'my_cookbook', path: 'cookbooks/my_cookbook'

Eventually, you can start your VM using Vagrant. Berkshelf will first download and then install all the required cookbooks in the Berkshelf, and upload them on the Chef server. Only after all the cookbooks are made available on the Chef server by Berkshelf, will Vagrant go on:

mma@mma-mbp:~/work/chef-repo $ vagrant up
Bringing machine 'server' up with 'virtualbox' provider...

==> default: Loading Berkshelf datafile...
==> default: Updating Vagrant's Berkshelf...
==> default: Resolving cookbook dependencies...
==> default: Fetching 'my_cookbook' from source at cookbooks/my_cookbook
==> default: Fetching cookbook index from https://supermarket.getchef.com...
...TRUNCATED OUTPUT...

This way, using Berkshelf together with Vagrant, saves a lot of manual steps and gets faster cycle times for your cookbook development.

See also

Downloading and integrating cookbooks as vendor branches into your Git repository

The Chef community offers a wide variety of ready-made cookbooks for many major software packages. They're a great starting point for your own infrastructure. However, usually you need to modify these cookbooks to suit your needs. Modifying your local copy of a community cookbook leaves you with the dilemma of not being able to update to the latest version of the community cookbook without losing your local changes.

Getting ready

You'll need to make sure that your local Git repository is clean and does not have any uncommitted changes:

mma@laptop:~/chef-repo $ git status
# On branch master
nothing to commit (working directory clean)

How to do it...

Carry out the following given steps:

  1. Go to https://supermarket.chef.io/cookbooks and search for the cookbook you need. In our example, we will use the mysql cookbook, which you can find by typing mysql in the search box and hitting enter. All we need is to note down the exact name of the cookbook; in this case, it's simply mysql.
    How to do it...
  2. Use knife to pull down the cookbook and integrate it with your local repository:
    mma@laptop:~/chef-repo $ knife cookbook site install mysql
    
    Installing mysql to /Users/mma/work/chef-repo/cookbooks
    …TRUNCATED OUTPUT…

    Verify the downloaded cookbooks:

    mma@laptop:~/chef-repo $ cd cookbooks
    mma@laptop:~/chef-repo/cookbooks $ ls -l
    
    total 8
    -rw-r--r--   1 mma  staff  3064 27 Sep  2013 README.md
    
    
    drwxr-xr-x+ 10 mma  staff   340  7 Dez 20:43 mysql
    drwxr-xr-x+ 12 mma  staff   408  7 Dez 20:43 yum
    drwxr-xr-x+  9 mma  staff   306  7 Dez 20:43 yum-mysql-community

    Validate the Git status:

    mma@laptop:~/chef-repo/cookbooks $ git status
    
    # On branch master
    # Your branch is ahead of 'origin/master' by 3 commits.
    #
    nothing to commit (working directory clean)
  3. You might have noticed that your local branch received three commits. Let's take a look at those:
    mma@laptop:~/chef-repo/cookbooks $ git log
    
    commit 271d3de3b95bdc32d68133cdc91cb04e09625f59
    Author: Matthias Marschall <mm@agileweboperations.com>
    Date:   Sun Dec 7 20:43:50 2014 +0100
    
        Import yum version 3.5.1
    
    commit cc72319ca7989df26e0ba7c3a76f78f6a7a7a4e2
    Author: Matthias Marschall <mm@agileweboperations.com>
    Date:   Sun Dec 7 20:43:44 2014 +0100
    
        Import yum-mysql-community version 0.1.10
    
    commit 30984edb00c12177e25558bdfcd519da508b3ac5
    Author: Matthias Marschall <mm@agileweboperations.com>
    Date:   Sun Dec 7 20:43:38 2014 +0100
    
        Import mysql version 5.6.1

The knife command successfully downloaded and imported the mysql cookbook as well as its dependencies.

How it works...

Knife executes a set of commands to download the desired cookbook and to integrate it with your local repository.

Let's take a look at the output of the knife cookbook site install command again and go through it step-by-step:

  1. First, the command makes sure that you're on the master branch of your repository:
    Checking out the master branch.
    
  2. The next step is to create a new vendor branch for the mysql cookbook, if none exists so far:
    Creating pristine copy branch chef-vendor-mysql.
    
  3. Then, knife downloads the tarball, removes any older version, uncompresses the new tarball, and removes it after successfully extracting its contents into a new cookbook directory:
    Downloading mysql from the cookbooks site at version 5.6.1 to /Users/mma/work/chef-repo/cookbooks/mysql.tar.gz
    Cookbook saved: /Users/mma/work/chef-repo/cookbooks/mysql.tar.gz
    Removing pre-existing version.
    Uncompressing mysql version 5.6.1.
    Removing downloaded tarball
  4. Now, it's time to commit the newly extracted files to the vendor branch:
    1 files updated, committing changes
  5. Finally, Git tags the branch with the current version of the cookbook:
    Creating tag cookbook-site-imported-mysql-5.6.1

The knife cookbook site install command executes all the previously mentioned steps for all the cookbooks the desired cookbook depends on, by default.

Eventually, you end up with a separate branch, the so-called vendor branch, for every downloaded cookbook integrated into your master branch and nicely tagged as shown:

mma@laptop:~/chef-repo$ git branch -a
  chef-vendor-iptables
  chef-vendor-mysql
* master
  remotes/origin/master

This approach enables you to change whatever you like in your master branch and still pull down newer versions of the community cookbook. Git will automatically merge both versions or ask you to remove conflicts manually; these are standard Git procedures.

There's more...

If you want to integrate the desired cookbook into another branch, use the --branch BRANCH_NAME parameter.

mma@laptop:~/chef-repo [experimental] $ knife cookbook site install mysql –-branch experimental
Installing mysql to /Users/mma/work/chef-repo/cookbooks
Checking out the experimental branch.
Pristine copy branch (chef-vendor-mysql) exists, switching to it.
Downloading mysql from the cookbooks site at version 5.6.1 to /Users/mma/work/chef-repo/cookbooks/mysql.tar.gz
Cookbook saved: /Users/mma/work/chef-repo/cookbooks/mysql.tar.gz
Removing pre-existing version.
Uncompressing mysql version 5.6.1.
removing downloaded tarball
No changes made to mysql
Checking out the experimental branch.
…TRUNCATED OUTPUT…

As you can see, instead of checking out the master branch, the knife cookbook site install command uses the experimental branch now.

You can use the -D switch when running the command in order to avoid downloading all the cookbooks that your desired cookbook depends on.

mma@laptop:~/chef-repo $ knife cookbook site install mysql -D
Installing mysql to /Users/mma/work/chef-repo/cookbooks
Checking out the master branch.
Pristine copy branch (chef-vendor-mysql) exists, switching to it.
Downloading mysql from the cookbooks site at version 5.6.1 to /Users/mma/work/chef-repo/cookbooks/mysql.tar.gz
Cookbook saved: /Users/mma/work/chef-repo/cookbooks/mysql.tar.gz
Removing pre-existing version.
Uncompressing mysql version 5.6.1.
removing downloaded tarball
No changes made to mysql
Checking out the master branch.

You can see that the command stopped after dealing with the mysql cookbook. It has not yet gotten the other cookbooks.

See also

  • You can use Berkshelf to manage cookbooks and their dependencies for you, which makes the preceding approach obsolete. See the Managing cookbook dependencies with Berkshelf recipe in this chapter.

Using custom knife plugins

Knife comes with a set of commands out of the box. The built-in commands deal with the basic elements of Chef-like cookbooks, roles, data bags, and so on. However, it would be nice to use knife for more than just the basic stuff. Fortunately, knife comes with a plugin API and there are already a host of useful knife plugins built by the makers of Chef and the Chef community.

Getting ready

Make sure you have an account at Amazon Web Services (AWS) if you want to follow along and try out the knife-ec2 plugin. There are knife plugins available for most Cloud providers. Go through the There's more... section of this recipe for the list.

How to do it...

Let's see which knife plugins are available, and try to use one to manage Amazon EC2 instances:

  1. List the knife plugins that are shipped as Ruby gems using the chef command-line tool:
    mma@laptop:~/chef-repo $ chef gem search -r knife-
    
    *** REMOTE GEMS ***
    ...TRUNCATED OUTPUT...
    
    knife-azure (1.3.0)
    ...TRUNCATED OUTPUT...
    knife-ec2 (0.10.0)
    ...TRUNCATED OUTPUT...
  2. Install the EC2 plugin to manage servers in the Amazon AWS Cloud:
    mma@laptop:~/chef-repo $ chef gem install knife-ec2
    
    Building native extensions.  This could take a while...
    ...TRUNCATED OUTPUT...
    Fetching: knife-ec2-0.10.0.gem (100%)
    Successfully installed knife-ec2-0.10.0
    ...TRUNCATED OUTPUT...
    
    6 gems installed
  3. List all the available instance types in AWS using the knife ec2 plugin. Please use your own AWS credentials instead of XXX and YYYYY:
    mma@laptop:~/chef-repo $ knife ec2 flavor list --aws-access-key-id XXX --aws-secret-access-key YYYYY
    
    ID           Name                                 Arch    RAM     Disk     Cores      
    c1.medium    High-CPU Medium                          32-bit  1740.8  350 GB   5          
    …TRUNCATED OUTPUT…
    m2.xlarge    High-Memory Extra Large                  64-bit  17510.  420 GB   6.5        
    t1.micro     Micro Instance                           0-bit   613     0 GB     2

How it works...

Knife looks for plugins at various places.

First, it looks into the .chef directory, which is located inside your current Chef repository, to find the plugins specific to this repository:

./.chef/plugins/knife/

Then, it looks into the .chef directory, which is located in your home directory, to find the plugins that you want to use in all your Chef repositories:

~/.chef/plugins/knife/

Finally, it looks for installed gems. Knife will load all the code from any chef/knife/ directory found in your installed Ruby gems. This is the most common way of using plugins developed by Chef or the Chef community.

There's more...

There are hundreds of knife plugins, including plugins for most of the major Cloud providers, as well as the major virtualization technologies, such as VMware, vSphere, and Openstack, amongst others.

See also

  • To learn how to write your own knife plugins, see the Creating custom knife plugins recipe in Chapter 2, Evaluating and Troubleshooting Cookbooks and Chef Runs
  • Find a list of supported Cloud providers at http://docs.chef.io/plugin_knife.html

Deleting a node from the Chef server

Bootstrapping a node not only installs Chef on that node, but creates a client object on the Chef server as well. Running the Chef client on your node uses the client object to authenticate itself against the Chef server on each run.

Additionally, to registering a client, a node object is created on the Chef server. The node object is the main data structure, which you can use to query node data inside your recipes.

Getting ready

Make sure you have at least one node registered on your Chef server that is safe to remove.

How to do it...

Let's delete the node and client object to completely remove a node from the Chef server.

  1. Delete the node object:
    mma@laptop:~/chef-repo $ knife node delete my_node
    
    Do you really want to delete my_node? (Y/N) y
    Deleted node[my_node]
  2. Delete the client object:
    mma@laptop:~/chef-repo $ knife client delete my_node
    
    Do you really want to delete my_node? (Y/N) y
    Deleted client[my_node]

How it works...

To keep your Chef server clean, it's important to not only manage your node objects but to also take care of your client objects, as well.

Knife connects to the Chef server and deletes the node object with a given name, using the Chef server RESTful API.

The same happens while deleting the client object on the Chef server.

After deleting both objects, your node is totally removed from the Chef server. Now, you can reuse the same node name with a new box or virtual machine.

There's more...

It is a bit tedious and error prone when you have to issue two commands. To simplify things, you can use a knife plugin called playground.

  1. Run the chef command-line tool to install the knife plugin:
    mma@laptop:~/chef-repo $ chef gem install knife-playground
    
    ...TRUNCATED OUTPUT...
    Installing knife-playground (0.2.2)
  2. Run the knife pg clientnode delete subcommand:
    mma@laptop:~/chef-repo $ knife pg clientnode delete my_node
    
    Deleting CLIENT my_node...
    Do you really want to delete my_node? (Y/N) y
    Deleted client[my_node]
    Deleting NODE my_node...
    Do you really want to delete my_node? (Y/N) y
    Deleted node[my_node]

See also

  • Read about how to do this when using Vagrant in the Managing virtual machines with Vagrant recipe in this recipe
  • Read about how to set up your Chef server and register your nodes in the Using the hosted Chef platform recipe in this chapter

Developing recipes with local mode

If running your own Chef server seems like an overkill and you're not comfortable with using the hosted Chef, you can use local mode to execute cookbooks.

Getting ready

  1. Create a cookbook named my_cookbook by running the following command:
    mma@laptop:~/chef-repo $ chef generate cookbook cookbooks/my_cookbook
    
    Compiling Cookbooks...
    Recipe: code_generator::cookbook
    ...TRUNCATED OUTPUT...
  2. Edit the default recipe of my_cookbook so that it creates a temporary file:
    mma@laptop:~/chef-repo $ subl cookbooks/my_cookbook/recipes/default.rb
    
    file "/tmp/local_mode.txt" do
        content "created by chef client local mode"
    end

How to do it...

Let's run my_cookbook on your local workstation using Chef client's local mode:

  1. Run the Chef client locally with my_cookbook in the run list:
    mma@laptop:~/chef-repo $ chef-client -z -o my_cookbook
    
    [2014-12-11T22:54:44+01:00] INFO: Starting chef-zero on host localhost, port 8889 with repository at repository at /Users/mma/work/chef-repo
    [2014-12-11T22:54:44+01:00] INFO: Forking chef instance to converge...
    Starting Chef Client, version 11.18.0.rc.1
    [2014-12-11T22:54:44+01:00] INFO: *** Chef 11.18.0.rc.1 ***
    [2014-12-11T22:54:44+01:00] INFO: Chef-client pid: 20179
    [2014-12-11T22:54:47+01:00] WARN: Run List override has been provided.
    [2014-12-11T22:54:47+01:00] WARN: Original Run List: []
    [2014-12-11T22:54:47+01:00] WARN: Overridden Run List: [recipe[my_cookbook]]
    [2014-12-11T22:54:47+01:00] INFO: Run List is [recipe[my_cookbook]]
    [2014-12-11T22:54:47+01:00] INFO: Run List expands to [my_cookbook]
    [2014-12-11T22:54:47+01:00] INFO: Starting Chef Run for webops
  2. Validate that the Chef client run creates the desired temporary file:
    mma@laptop:~/chef-repo $ cat /tmp/local_mode.txt
    
    created by chef client local mode%

How it works...

The -z parameter switches the Chef client into local mode. Local mode uses chef-zero—a simple, in-memory version of the Chef server provided by Chef DK—when converging the local workstation.

By providing the -o parameter, you override the run list of your local node so that the Chef client executes the default recipe from my_cookbook.

There's more...

Chef-zero saves all modifications made by your recipes to the local file system. It creates a JSON file containing all node attributes for your local workstation in the nodes directory. This way, the next time you run the Chef client in local mode, it will be aware of any changes your recipes made to the node.

Running knife in local mode

You can use knife in local mode, too. To set the run list of your node named laptop (instead of having to override it with -o), you can run the following command:

mma@laptop:~/chef-repo $ knife node run_list add -z laptop 'recipe[my_cookbook]'

Moving to hosted Chef or your own Chef server

When you're done editing and testing your cookbooks on your local workstation with chef-zero, you can seamlessly upload them to hosted Chef or your own Chef server:

mma@laptop:~/chef-repo $ knife upload /
laptop:
  run_list: recipe[my_cookbook]

See also

Using roles

Roles are there in Chef to group nodes with similar configuration. Typical cases are to have roles for web servers, database servers, and so on.

You can set custom run lists for all the nodes in your roles and override attribute values from within your roles.

Let's see how to create a simple role.

Getting ready

For the following examples, I assume that you have a node named server and that you have at least one cookbook (I'll use the ntp cookbook) registered with your Chef server.

How to do it...

Let's create a role and see what we can do with it.

  1. Create a role:
    mma@laptop:~/chef-repo $ subl roles/web_servers.rb
    
    name "web_servers"
    description "This role contains nodes, which act as web servers"
    run_list "recipe[ntp]"
    default_attributes 'ntp' => {
      'ntpdate' => {
        'disable' => true
      }
    }
  2. Upload the role on the Chef server:
    mma@laptop:~/chef-repo $ knife role from file web_servers.rb
    
    Updated Role web_servers!
  3. Assign the role to a node called server:
    mma@laptop:~/chef-repo $ knife node run_list add server 'role[web_servers]'
    
    server:
      run_list: role[web_servers]
  4. Run the Chef client:
    user@server:~$ sudo chef-client
    
    ...TRUNCATED OUTPUT...
    [2014-12-25T13:28:24+00:00] INFO: Run List is [role[web_servers]]
    [2014-12-25T13:28:24+00:00] INFO: Run List expands to [ntp]
    ...TRUNCATED OUTPUT...

How it works...

You define a role in a Ruby file inside the roles folder of your Chef repository. A role consists of a name attribute and a description attribute. Additionally, a role usually contains a role-specific run list and role-specific attribute settings.

Every node, which has a role in its run list, will have the role's run list expanded into its own. This means that all the recipes (and roles), which are in the role's run list, will be executed on your nodes.

You need to upload your role on your Chef server by using the knife role from file command.

Only then should you add the role to your node's run list.

Running the Chef client on a node having your role in its run list will execute all the recipes listed in the role.

The attributes you define in your role will be merged with attributes from environments and cookbooks, according to the precedence rules described at https://docs.chef.io/roles.html#attribute-precedence.

See also

  • Find out how roles can help you in finding nodes in the Using search to find nodes recipe in Chapter 4, Writing Better Cookbooks
  • Learn more about in the Overriding attributes recipe in Chapter 4, Writing Better Cookbooks
  • Read everything about roles at https://docs.chef.io/roles.html

Using environments

Having separate environments for development, testing, and production are good ways to be able to develop and test cookbook updates and other configuration changes in isolation. Chef enables you to group your nodes into separate environments so as to support an ordered development flow.

Getting ready

For the following examples, I assume that you have a node named server in the _default environment and that you have at least one cookbook (I'll use the ntp cookbook) registered with your Chef server.

How to do it...

Let's see how to manipulate environments using knife.

Note

This is only a good idea if you want to play around. For serious work, please create files describing your environments and put them under version control as described in the There's more... section of this recipe.

  1. Create your environment on the fly using knife. The following command will open your shell's default editor so that you can modify the environment definition:

    Tip

    Make sure you've set your EDITOR environment variable to your preferred one.

    mma@laptop:~/chef-repo $ knife environment create book
    
    {
      "name": "book",
      "description": "",
      "cookbook_versions": {
      },
      "json_class": "Chef::Environment",
      "chef_type": "environment",
      "default_attributes": {
      },
      "override_attributes": {
      }
    }
    Created book
  2. List the available environments:
    mma@laptop:~/chef-repo $ knife environment list
    
    _default
    book
  3. List the nodes for all the environments:
    mma@laptop:~/chef-repo $ knife node list
    
    server
  4. Verify that the node server is not in the book environment yet by listing nodes in the book environment only:
    mma@laptop:~/chef-repo $ knife node list -E book
    mma@laptop:~/chef-repo $
    
  5. Change the environment of server to book using knife:
    mma@laptop:~/chef-repo $ knife node environment set server book
    
    server:
      chef_environment: book
  6. List the nodes of the book environment again:
    mma@laptop:~/chef-repo $ knife node list -E book
    
    server
  7. Use specific cookbook versions and override certain attributes for the environment:
    mma@laptop:~/chef-repo $ knife environment edit book
    
    {
      "name": "book",
      "description": "",
      "cookbook_versions": {
        "ntp": "1.6.8"
      },
      "json_class": "Chef::Environment",
      "chef_type": "environment",
      "default_attributes": {
      },
      "override_attributes": {
        "ntp": {
          "servers": ["0.europe.pool.ntp.org", "1.europe.pool.ntp.org", "2.europe.pool.ntp.org", "3.europe.pool.ntp.org"]
        }
      }
    }
    Saved book

How it works...

A common use of environments is to promote cookbook updates from development to staging and then into production. Additionally, they enable you to use different cookbook versions on separate sets of nodes and environment-specific attributes. You might have nodes with lesser memory in your staging environment as in your production environment. By using environment-specific default attributes, you can, for example, configure your MySQL service to consume lesser memory on staging than on production.

Note

The Chef server always has an environment called _default, which cannot be edited or deleted. All the nodes go in there if you don't specify any other environment.

Be aware that roles are not environment-specific. You may use environment-specific run lists, though.

The node's environment can be queried using the node.chef_environment method inside your cookbooks.

There's more...

If you want your environments to be under version control (and you should), a better way to create a new environment is to create a new Ruby file in the environments directory inside your Chef repository:

mma@laptop:~/chef-repo $ cd environments
mma@laptop:~/chef-repo $ subl book.rb
name "book"

You should add, commit, and push your new environment file to GitHub:

mma@laptop:~/chef-repo $ git add environments/book.rb
mma@laptop:~/chef-repo $ git commit -a -m "the book env"
mma@laptop:~/chef-repo $ git push

Now, can create the environment on the Chef server from the newly created file using knife:

mma@laptop:~/chef-repo $ knife environment from file book.rb
Created Environment book

Tip

You have to deal with two artifact storages here. You have to use your version control system and knife / Berkshelf to sync your local changes to your Chef server. The Chef server is not aware of any changes that you do when using your version control system and vice versa.

There is a way to migrate all the nodes from one environment to another by using knife exec:

mma@laptop:~/chef-repo $ knife exec -E 'nodes.transform("chef_environment:_default") { |n| n.chef_environment("book")

You can limit your search for nodes in a specific environment:

mma@laptop:~/chef-repo $ knife search node "chef_environment:book"
1 item found

See also

  • If you want to set up a virtual machine as a node, see the Managing virtual machines with Vagrant recipe in this chapter
  • Read more about environments at https://docs.chef.io/environments.html

Freezing cookbooks

Uploading broken cookbooks that override your working ones is a major pain and can result in widespread outage throughout your infrastructure. If you have a cookbook version that is known to work, it's a good idea to freeze this version so that no one can overwrite the same version with broken code. When used together with version constraints that are specified in your environment manifests, freezing cookbooks can keep your production servers safe from accidental changes.

Getting ready

Make sure you have at least one cookbook (I'll use the ntp cookbook) registered with your Chef server.

How to do it...

Let's see what happens if we freeze a cookbook.

  1. Upload a cookbook and freeze it:
    mma@laptop:~/chef-repo $ knife cookbook upload ntp --freeze
    
    Uploading ntp            [1.6.8]
    Uploaded 1 cookbook.
  2. Try to upload the same cookbook version again:
    mma@laptop:~/chef-repo $ knife cookbook upload ntp
    
    Uploading ntp            [1.6.8]
    ERROR: Version 1.6.8 of cookbook ntp is frozen. Use --force to override.
    WARNING: Not updating version constraints for ntp in the environment as the cookbook is frozen.
    ERROR: Failed to upload 1 cookbook.
  3. Change the cookbook version:
    mma@laptop:~/chef-repo $ subl cookbooks/ntp/metadata.rb
    
    …
    version           "1.6.9"
  4. Upload the cookbook again:
    mma@laptop:~/chef-repo $ knife cookbook upload ntp
    
    Uploading ntp            [1.6.9]
    Uploaded 1 cookbook.

How it works...

By using the --freeze option when uploading a cookbook, you tell the Chef server that it should not accept any changes to the same version of the cookbook anymore. This is important if you're using environments and want to make sure that your production environment cannot be broken by uploading a corrupted cookbook.

By changing the version number of your cookbook, you can upload the new version. Then you can make, for example, your staging environment use that new cookbook version.

There's more...

To support a more elaborate workflow, you can use the knife-spork knife plugin. It helps multiple developers work on the same Chef server and repository without treading on each other's toes. You can find more information about it at https://github.com/jonlives/knife-spork.

See also

Running Chef client as a daemon

While you can run the Chef client on your nodes manually whenever you change something in your Chef repository, it's sometimes preferable to have the Chef client run automatically every so often. Letting the Chef client run automatically makes sure that no box misses out any updates.

Getting ready

You need to have a node registered with your Chef server. It needs to be able to run chef-client without any errors.

How to do it...

Let's see how to start the Chef client in the daemon mode so that it runs automatically.

  1. Start the Chef client in the daemon mode, running every 30 minutes:
    user@server:~$ sudo chef-client -i 1800
    
  2. Validate that the Chef client runs as daemon:
    user@server:~$ ps auxw | grep chef-client
    

How it works...

The -i parameter will start the Chef client as a daemon. The given number is the seconds between each Chef client run. In the previous example, we specified 1,800 seconds, which results in the Chef client running every 30 minutes.

You can use the same command in a service startup script.

There's more...

Instead of running the Chef client as a daemon, you can use a Cronjob to run it every so often:

user@server:~$ subl /etc/cron.d/chef_client
PATH=/usr/local/bin:/usr/bin:/bin
# m h dom mon dow user command
*/15 * * * * root chef-client -l warn | grep -v 'retrying [1234]/5 in'

This cronjob will run the Chef client every 15 minutes and swallow the first four retrying warning messages. This is important to avoid Cron sending out e-mails if the Chef server is a little slow and the Chef client needs a few retries.

Note

It is possible to initiate a Chef client run at any time by sending the SIGUSR1 signal to the Chef client daemon:

user@server:~$ sudo killall -USR1 chef-client

Using chef-shell

Writing cookbooks is hard. Being able to try out parts of a recipe interactively and using breakpoints really helps to understand how your recipes work.

Chef comes with chef-shell, which is essentially an interactive Ruby session with Chef. In chef-shell, you can create attributes, write recipes, and initialize Chef runs, among other things. Chef-shell allows you to evaluate parts of your recipes on the fly before uploading them to your Chef server and executeing complete cookbooks on your nodes.

How to do it...

Running chef-shell is straightforward.

  1. Start chef-shell in standalone mode:
    mma@laptop:~/chef-repo $ chef-shell
    loading configuration: none (standalone chef-shell session)
    Session type: standalone
    Loading......done.
    
    
    This is the chef-shell.
     Chef Version: 11.18.0
     http://www.chef.io/chef
     http://docs.chef.io/
    
    run `help' for help, `exit' or ^D to quit.
    
    Ohai2u mma@laptop!
    chef >
    
  2. Switch to the attributes mode in chef-shell:
    chef > attributes_mode
    
  3. Set an attribute value to be used inside the recipe later:
    chef:attributes > set[:title] = "Chef Cookbook"
     => "Chef Cookbook"
    chef:attributes > quit
     => :attributes
    chef >
    
  4. Switch to the recipe mode:
    chef > recipe_mode
    
  5. Create a file resource inside a recipe, using the title attribute as content:
    chef:recipe > file "/tmp/book.txt" do
    chef:recipe >     content node.title
    chef:recipe ?> end
     => <file[/tmp/book.txt] @name: "/tmp/book.txt" @noop: nil @before: nil @params: {} @provider: Chef::Provider::File @allowed_actions: [:nothing, :create, :delete, :touch, :create_if_missing] @action: "create" @updated: false @updated_by_last_action: false @supports: {} @ignore_failure: false @retries: 0 @retry_delay: 2 @source_line: "(irb#1):1:in `irb_binding'" @elapsed_time: 0 @resource_name: :file @path: "/tmp/book.txt" @backup: 5 @diff: nil @cookbook_name: nil @recipe_name: nil @content: "Chef Cookbook">
    chef:recipe >
    
  6. Initiate a Chef run to create the file with the given content:
    chef:recipe > run_chef
    
    [2014-12-12T22:26:42+01:00] INFO: Processing file[/tmp/book.txt] action create ((irb#1) line 1)
    ...TRUNCATED OUTPUT...
    => true

How it works...

Chef-shell starts an interactive Ruby Shell (IRB) session, which is enhanced with some Chef-specific features. It offers certain modes, such as attributes_mode or recipe_mode, which enable you to write commands like you would put them into attributes file or recipes.

Entering a resource command into the recipe context will create the given resource, but not run it yet. It's like Chef reading your recipe files and creating the resources but not yet running them. You can run all the resources you created within the recipe context using the run_chef command. This will execute all the resources on your local box and physically change your system. To play around with temporary files, your local box might do, but if you're going to do more invasive stuff, such as installing or removing packages, installing services, and so on, you might want to use chef-shell from within a Vagrant VM.

There's more...

Not only can you run chef-shell in standalone mode but you can also in Chef client mode. If you run it in Chef client mode, it will load the complete run list of your node and you'll be able to tweak it inside the chef-shell. You start the Chef client mode by using the run it --client parameter:

mma@laptop:~/chef-repo $ chef-shell --client

You can configure which Chef server to connect it to in a file called chef-shell.rb, in the same way as you do in the client.rb file on your local workstation.

You can use chef-shell to manage your Chef server, for example, listing all nodes:

chef > nodes.list
[node[my_server]]

You can put breakpoints into your recipes. If it hits a breakpoint resource, chef-shell will stop the execution of the recipe and you'll be able to inspect the current state of your Chef run:

breakpoint "name" do
  action :break
end

See also

Left arrow icon Right arrow icon

Description

This book is for system engineers and administrators who have a fundamental understanding of information management systems and infrastructure. It helps if you've already played around with Chef; however, this book covers all the important topics you will need to know. If you don't want to dig through a whole book before you can get started, this book is for you, as it features a set of independent recipes you can try out immediately.

What you will learn

  • Set up your local development and testing environment for Chef
  • Debug your cookbooks and Chef runs by using the numerous inspection and logging facilities of Chef
  • Drive your cookbooks from external data or nodespecific attributes
  • Manage and scale your cloud infrastructure by automating your configuration management
  • Extend Chef to meet your advanced needs by creating custom plugins for knife and Ohai
  • Test your Chef cookbooks and infrastructure by writing examples

Product Details

Country selected
Publication date, Length, Edition, Language, ISBN-13
Publication date : May 29, 2015
Length: 278 pages
Edition : 1st
Language : English
ISBN-13 : 9781785287947
Vendor :
Chef
Tools :

What do you get with a Packt Subscription?

Free for first 7 days. €18.99 p/m after that. Cancel any time!
Product feature icon Unlimited ad-free access to the largest independent learning library in tech. Access this title and thousands more!
Product feature icon 50+ new titles added per month, including many first-to-market concepts and exclusive early access to books as they are being written.
Product feature icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Product feature icon Thousands of reference materials covering every tech concept you need to stay up to date.
Subscribe now
View plans & pricing

Product Details

Publication date : May 29, 2015
Length: 278 pages
Edition : 1st
Language : English
ISBN-13 : 9781785287947
Vendor :
Chef
Tools :

Packt Subscriptions

See our plans and pricing
Modal Close icon
€18.99 billed monthly
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Simple pricing, no contract
€189.99 billed annually
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just €5 each
Feature tick icon Exclusive print discounts
€264.99 billed in 18 months
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just €5 each
Feature tick icon Exclusive print discounts

Frequently bought together


Stars icon
Total 73.98
Chef Infrastructure Automation Cookbook Second Edition
€36.99
Mastering Chef
€36.99
Total 73.98 Stars icon

Table of Contents

8 Chapters
1. Chef Infrastructure Chevron down icon Chevron up icon
2. Evaluating and Troubleshooting Cookbooks and Chef Runs Chevron down icon Chevron up icon
3. Chef Language and Style Chevron down icon Chevron up icon
4. Writing Better Cookbooks Chevron down icon Chevron up icon
5. Working with Files and Packages Chevron down icon Chevron up icon
6. Users and Applications Chevron down icon Chevron up icon
7. Servers and Cloud Infrastructure Chevron down icon Chevron up icon
Index Chevron down icon Chevron up icon

Customer reviews

Top Reviews
Rating distribution
Full star icon Full star icon Full star icon Half star icon Empty star icon 3.7
(6 Ratings)
5 star 50%
4 star 16.7%
3 star 0%
2 star 16.7%
1 star 16.7%
Filter icon Filter
Top Reviews

Filter reviews by




Michael Aug 08, 2015
Full star icon Full star icon Full star icon Full star icon Full star icon 5
I really like this book so far. I was concerned that the recipe format would not be best for a beginner but it has great step-by-step guides in here that will bring you up to speed quickly. Highly recommend for those just learning Chef.
Amazon Verified review Amazon
John D. Aug 09, 2015
Full star icon Full star icon Full star icon Full star icon Full star icon 5
It helps to have a basic understanding of how Chef works before this book is really valuable, and this book could be summed up as chapters full of bite-sized mini-HOWTOs that help you get from basic knowledge to really knowing your way around.I use Chef a lot, and really enjoyed reading the first edition of this book, and I hadn't used Chef a whole ton back then. Upon reading the second edition, I'm still able to have some "a-ha" moments, and further my understanding. Another great way to think about this book is a substitute for the Chef documentation - it's a great cheat sheet for some of the most common tasks you'll perform.Some of the only quirks (should only impact novice to intermediate users):- the lack of visual representations, but this will matter a lot less if you have some experience- the mini-HOWTO (my description) sections have 'how it works' subsections at the end - you may want to skip ahead if things don't make sense- it would also really help to have a basic understanding of git and vagrant before startingSide note: this is in the upper 5% of Packt Publishing books I've readTL;DR: a must-have resource for any Chef user, and the only books that compare in any way are the O'Reilly books on Chef
Amazon Verified review Amazon
Antonio Dec 22, 2015
Full star icon Full star icon Full star icon Full star icon Full star icon 5
Very helpful reading. I have worked with Ansible an Salt before and this book got me up to speed with Chef. Of course, it's not deeply elaborating specific points but if you already have knowledge of configuration management this will help you to catch up quickly.
Amazon Verified review Amazon
Jean Remy Aug 05, 2015
Full star icon Full star icon Full star icon Full star icon Empty star icon 4
This book has a lot of great examples which helped me tremendously testing some of the recipes. As an intermediate user it was fairly easy to work my way through the chapters. However, this book is not for a novice user since it does not have a lot of diagrams to explain visually the overall intent of each recipes.
Amazon Verified review Amazon
Matthew Dresden Dec 21, 2016
Full star icon Full star icon Empty star icon Empty star icon Empty star icon 2
This is too far out of date to be of any practical use, but unfortunately this is the case for all Chef books.Chefs docs and Github are the only place worth looking at, plus someone who already knows chef.It would otherwise be a very long painful process to real use chef appropriately.It advises against wrapping, which is now the standard.
Amazon Verified review Amazon
Get free access to Packt library with over 7500+ books and video courses for 7 days!
Start Free Trial

FAQs

What is included in a Packt subscription? Chevron down icon Chevron up icon

A subscription provides you with full access to view all Packt and licnesed content online, this includes exclusive access to Early Access titles. Depending on the tier chosen you can also earn credits and discounts to use for owning content

How can I cancel my subscription? Chevron down icon Chevron up icon

To cancel your subscription with us simply go to the account page - found in the top right of the page or at https://subscription.packtpub.com/my-account/subscription - From here you will see the ‘cancel subscription’ button in the grey box with your subscription information in.

What are credits? Chevron down icon Chevron up icon

Credits can be earned from reading 40 section of any title within the payment cycle - a month starting from the day of subscription payment. You also earn a Credit every month if you subscribe to our annual or 18 month plans. Credits can be used to buy books DRM free, the same way that you would pay for a book. Your credits can be found in the subscription homepage - subscription.packtpub.com - clicking on ‘the my’ library dropdown and selecting ‘credits’.

What happens if an Early Access Course is cancelled? Chevron down icon Chevron up icon

Projects are rarely cancelled, but sometimes it's unavoidable. If an Early Access course is cancelled or excessively delayed, you can exchange your purchase for another course. For further details, please contact us here.

Where can I send feedback about an Early Access title? Chevron down icon Chevron up icon

If you have any feedback about the product you're reading, or Early Access in general, then please fill out a contact form here and we'll make sure the feedback gets to the right team. 

Can I download the code files for Early Access titles? Chevron down icon Chevron up icon

We try to ensure that all books in Early Access have code available to use, download, and fork on GitHub. This helps us be more agile in the development of the book, and helps keep the often changing code base of new versions and new technologies as up to date as possible. Unfortunately, however, there will be rare cases when it is not possible for us to have downloadable code samples available until publication.

When we publish the book, the code files will also be available to download from the Packt website.

How accurate is the publication date? Chevron down icon Chevron up icon

The publication date is as accurate as we can be at any point in the project. Unfortunately, delays can happen. Often those delays are out of our control, such as changes to the technology code base or delays in the tech release. We do our best to give you an accurate estimate of the publication date at any given time, and as more chapters are delivered, the more accurate the delivery date will become.

How will I know when new chapters are ready? Chevron down icon Chevron up icon

We'll let you know every time there has been an update to a course that you've bought in Early Access. You'll get an email to let you know there has been a new chapter, or a change to a previous chapter. The new chapters are automatically added to your account, so you can also check back there any time you're ready and download or read them online.

I am a Packt subscriber, do I get Early Access? Chevron down icon Chevron up icon

Yes, all Early Access content is fully available through your subscription. You will need to have a paid for or active trial subscription in order to access all titles.

How is Early Access delivered? Chevron down icon Chevron up icon

Early Access is currently only available as a PDF or through our online reader. As we make changes or add new chapters, the files in your Packt account will be updated so you can download them again or view them online immediately.

How do I buy Early Access content? Chevron down icon Chevron up icon

Early Access is a way of us getting our content to you quicker, but the method of buying the Early Access course is still the same. Just find the course you want to buy, go through the check-out steps, and you’ll get a confirmation email from us with information and a link to the relevant Early Access courses.

What is Early Access? Chevron down icon Chevron up icon

Keeping up to date with the latest technology is difficult; new versions, new frameworks, new techniques. This feature gives you a head-start to our content, as it's being created. With Early Access you'll receive each chapter as it's written, and get regular updates throughout the product's development, as well as the final course as soon as it's ready.We created Early Access as a means of giving you the information you need, as soon as it's available. As we go through the process of developing a course, 99% of it can be ready but we can't publish until that last 1% falls in to place. Early Access helps to unlock the potential of our content early, to help you start your learning when you need it most. You not only get access to every chapter as it's delivered, edited, and updated, but you'll also get the finalized, DRM-free product to download in any format you want when it's published. As a member of Packt, you'll also be eligible for our exclusive offers, including a free course every day, and discounts on new and popular titles.