Conquer by dividing

Depending on the size of your deployment and the way you connect to all your nodes, a masterless solution may be a good fit. In a masterless configuration, you don't run the Puppet agent; rather, you push the Puppet code to a node, and then run Puppet apply. There are a few benefits to this method and a few drawbacks.

Benefits	Drawbacks
No single point of failure	Can't use built-in reporting tools such as dashboard.
Simpler configuration	Exported resources requires nodes have write access to the database.
Finer-grained control on where code is deployed	Each node has access to all the code
Multiple simultaneous runs do not affect each other (reduces contention)	More difficult to know when a node is failing to apply catalog correctly
Connection to Puppet master not required (offline possible)	No certificate management
No certificate management

The idea with a masterless configuration is that you distribute the Puppet code to each node individually and then kick off a puppet run to apply that code. One of the benefits of Puppet is that it keeps your system in a known good state, so when choosing masterless it is important to build your solution with this in mind. A cron job configured by your deployment mechanism that can apply Puppet to the node on a routine schedule will suffice.

The key parts of a masterless configuration are: distributing the code, pushing updates to the code, and ensuring the code is applied routinely to the nodes. Pushing a bunch of files to a machine is best done with some sort of package management.

Tip

Many masterless configurations use Git to have clients pull the files, this has the advantage of clients pulling changes.

For Linux systems, the big players are rpm and dpkg, whereas for MacOS, Installer package files can be used. It is also possible to configure the nodes to download the code themselves from a web location. Some large installations use Git to update the code as well.

The solution I will outline is that of using an rpm deployed through yum to install and run Puppet on a node. Once deployed, we can have the nodes pull updated code from a central repository rather than rebuild the rpm for every change.

Creating an rpm

To start our rpm, we will make an rpm spec file, we can make this anywhere since we don't have a master in this example. Start by installing rpm-build, which will allow us to build the rpm.

# yum install rpm-build
Installing
  rpm-build-4.8.0-37.el6.x86_64

It will be important later to have a user to manage the repository, so create a user called builder at this point. We'll do this on the Puppet master machine we built earlier. Create an rpmbuild directory with the appropriate subdirectories, and then create our example code in this location.

# sudo -iu builder
$ mkdir -p rpmbuild/{SPECS,SOURCES}
$ cd SOURCES
$ mkdir -p modules/example/manifests
$ cat <<EOF>modules/example/manifests/init.pp
class example {
notify {"This is an example.": }
file {'/tmp/example':
mode => '0644',
owner => '0',
group => '0',
content => 'This is also an example.'
}
}
EOF
$ tar cjf example.com-puppet-1.0.tar.bz2 modules

Next, create a spec file for our rpm in rpmbuild/SPECS as shown in the following commands:

Name:           example.com-puppet
Version: 1.0
Release: 1%{?dist}
Summary: Puppet Apply for example.com

Group: System/Utilities
License: GNU
Source0: example.com-puppet-%{version}.tar.bz2
BuildRoot: %(mktemp -ud %{_tmppath}/%{name}-%{version}-%{release}-XXXXXX)

Requires: puppet
BuildArch:      noarch

%description
This package installs example.com's puppet configuration
and applies that configuration on the machine.


%prep

%setup -q -c
%install
mkdir -p $RPM_BUILD_ROOT/%{_localstatedir}/local/puppet
cp -a . $RPM_BUILD_ROOT/%{_localstatedir}/local/puppet

%clean
rm -rf %{buildroot}

%files
%defattr(-,root,root,-)
%{_localstatedir}/local/puppet

%post
# run puppet apply
/bin/env puppet apply --logdest syslog --modulepath=%{_localstatedir}/local/puppet/modules %{_localstatedir}/local/puppet/manifests/site.pp 

%changelog
* Fri Dec 6 2013 Thomas Uphill <[e-mail]> - 1.0-1
- initial build

Then use rpmbuild to build the rpm based on this spec, as shown in the following command:

$ rpmbuild -ba example.com-puppet.spec
…
Wrote: /home/builder/rpmbuild/SRPMS/example.com-puppet-1.0-1.el6.src.rpm
Wrote: /home/builder/rpmbuild/RPMS/noarch/example.com-puppet-1.0-1.el6.noarch.rpm

Now, deploy a node and copy the rpm onto that node. Verify that the node installs Puppet and then does a Puppet apply run.

# yum install example.com-puppet-1.0-1.el6.noarch.rpm 
Loaded plugins: downloadonly
…
Installed:
  example.com-puppet.noarch 0:1.0-1.el6 
Dependency Installed:
  augeas-libs.x86_64 0:1.0.0-5.el6
...
  puppet-3.3.2-1.el6.noarch
…
Complete!

Verify that the file we specified in our package has been created by using the following command:

# cat /tmp/example
This is also an example.

Now, if we are going to rely on this system of pushing Puppet to nodes, we have to make sure we can update the rpm on the clients and we have to ensure that the nodes still run Puppet regularly so as to avoid configuration drift (the whole point of Puppet). There are many ways to accomplish these two tasks. We can put the cron definition into the post section of our rpm:

%post
# install cron job
/bin/env puppet resource cron 'example.com-puppet' command='/bin/env puppet apply --logdest syslog --modulepath=%{_localstatedir}/local/puppet/modules %{_localstatedir}/local/puppet/manifests/site.pp' minute='*/30' ensure='present'

We could have a cron job be part of our site.pp, as shown in the following command:

cron { 'example.com-puppet':
  ensure      => 'present',
  command => '/bin/env puppet apply --logdest syslog --modulepath=/var/local/puppet/modules /var/local/puppet/manifests/site.pp',
  minute  => ['*/30'],
  target   => 'root',
  user  => 'root',
}

To ensure the nodes have the latest version of the code, we can define our package in the site.pp.

package {'example.com-puppet':  ensure => 'latest' }

In order for that to work as expected, we need to have a yum repository for the package and have the nodes looking at that repository for packages.

Creating the YUM repository

Creating a YUM repository is a very straightforward task. Install the createrepo rpm and then run createrepo on each directory you wish to make into a repository.

# mkdir /var/www/html/puppet
# yum install createrepo
…
Installed:
 createrepo.noarch 0:0.9.9-18.el6   
# chown builder /var/www/html/puppet
# sudo -iu builder
$ mkdir /var/www/html/puppet/{noarch,SRPMS}
$ cp /home/builder/rpmbuild/RPMS/noarch/example.com-puppet-1.0-1.el6.noarch.rpm /var/www/html/puppet/noarch
$ cp rpmbuild/SRPMS/example.com-puppet-1.0-1.el6.src.rpm /var/www/html/puppet/SRPMS
$ cd /var/www/html/puppet
$ createrepo noarch
$ createrepo SRPMS

Our repository is ready, but we need to export it with the web server to make it available to our nodes. This rpm contains all our Puppet code, so we need to ensure that only the clients we wish get access to the files. We'll create a simple listener on port 80 for our Puppet repository

Listen 80
<VirtualHost *:80>
  DocumentRoot /var/www/html/puppet
</VirtualHost>

Now, the nodes need to have the repository defined on them so they can download the updates when they are made available via the repository. The idea here is that we push the rpm to the nodes and have them install the rpm. Once the rpm is installed, the yum repository pointing to updates is defined and the nodes continue updating themselves.

yumrepo { 'example.com-puppet':
  baseurl  => 'http://puppet.example.com/noarch',
  descr    => 'example.com Puppet Code Repository',
  enabled  => '1',
  gpgcheck => '0',
}

So to ensure that our nodes operate properly, we have to make sure of the following things:

• Install code
• Define repository
• Define cron job to run Puppet apply routinely
• Define package with latest tag to ensure it is updated

A default node in our masterless configuration requires that the cron task and the repository be defined. If you wish to segregate your nodes into different production zones (such as development, production, and sandbox), I would use a repository management system like Pulp. Pulp allows you to define repositories based on other repositories and keeps all your repositories consistent.

Tip

You should also setup a gpg key on the builder account that can sign the packages it creates. You would then distribute the gpg public key to all your nodes and enable gpgcheck on the repository definition.