When you start to think about dividing up your Puppet server, the main thing to realize is that many parts of Puppet are simply HTTP SSL transactions. If you treat those things as you would a web service, you can scale out to any size required using HTTP load balancing techniques.
The first step in splitting up the Puppet master is to configure the Puppet master to run under passenger. To ensure we all have the same infrastructure, we'll install a stock passenger configuration together and then start tweaking the configuration. We'll begin building on an x86_64 Enterprise 6 rpm-based Linux; the examples in this book were built using CentOS 6.5 and Springdale Linux 6.5 distributions. Once we have passenger running, we'll look at splitting up the workload.
In our example installation, we will be using the name puppet.example.com for our Puppet server. Starting with a server installation of Enterprise Linux version 6, we install httpd and mod_ssl using the following code:
Tip
Downloading the example code
You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
Note
In each example, I will install the latest available version for Enterprise Linux 6.5 and display the version for the package requested (some packages may pull in dependencies—those versions are not shown).
To install mod_passenger, we pull in the Extra Packages for Enterprise Linux (EPEL) repository available at https://fedoraproject.org/wiki/EPEL. Install the EPEL repository by downloading the rpm file from http://download.fedoraproject.org/pub/epel/6/x86_64/repoview/epel-release.html or use the following code:
Once EPEL is installed, we install mod_passenger from that repository using the following code:
Next, we will pull in Puppet from the puppetlabs repository available at http://docs.puppetlabs.com/guides/puppetlabs_package_repositories.html#for-red-hat-enterprise-linux-and-derivatives using the following code:
With the puppetlabs repository installed, we can then install Puppet using the following command:
The Puppet rpm will create the /etc/puppet and /var/lib/puppet directories. In /etc/puppet, there will be a template puppet.conf; we begin by editing that file to set the name of our Puppet server (puppet.example.com) in the certname setting using the following code:
The other lines in this file are defaults. At this point, we would expect puppet.example.com to be resolved with a DNS query correctly, but if you do not control DNS at your organization or cannot have this name resolved properly at this point, edit /etc/hosts, and put in an entry for your host pointing to puppet.example.com. In all the examples, you would substitute example.com for your own domain name.
We now need to create certificates for our master; to ensure the Certificate Authority (CA) certificates are created, run Puppet cert list using the following command:
In your enterprise, you may have to answer requests from multiple DNS names, for example, puppet.example.com, puppet, and puppet.devel.example.com. To make sure our certificate is valid for all those DNS names, we will pass the dns-alt-names option to puppet certificate generate; we also need to specify that the certificates are to be signed by the local machine using the following command:
Now, to sign the certificate request, first verify the certificate list using the following commands:
Tip
We specified the ssldir directive in our configuration. To interactively determine where the certificates will be stored using the following command line:
One last task is to copy the certificate that you just signed into certs by navigating to /var/lib/puppet/ssl/certs. You can use Puppet certificate find to do this using the following command:
When you install Puppet from the puppetlabs repository, the rpm will create an Apache configuration file called apache2.conf. Locate this file and copy it into your Apache configuration directory using the following command:
We will now show the Apache config file and point out the important settings using the following configuration:
The preceding lines of code configure passenger for performance. PassengerHighPerformance turns off some compatibility that isn't required. The other options are tuning parameters. For more information on these settings, see http://www.modrails.com/documentation/Users%20guide%20Apache.html.
Next we will need to modify the file to ensure it points to the newly created certificates. We will need to edit the lines for SSLCertificateFile and SSLCertificateKeyFile. The other SSL file settings should point to the correct certificate, chain, and revocation list files as shown in the following code:
In this VirtualHost we listen on 8140 and configure the SSL certificates in the SSL lines. The RequestHeader lines are used to pass certificate information to the Puppet process spawned by passenger. The DocumentRoot and RackBaseURI settings are used to tell passenger where to find its configuration file config.ru. We create /etc/puppet/rack and it's subdirectories and then copy the example config.ru into that directory using the following commands:
We change the owner of config.ru to puppet:puppet as the passenger process will run as the owner of config.ru. Our config.ru will contain the following code:
Tip
In this example, we have used the repository rpms supplied by Puppet and EPEL. In a production installation, you would use reposync to copy these repositories locally so that your Puppet machines do not need to access the Internet directly.
The config.ru file sets the command-line arguments for Puppet. The ARGV lines are used to set additional parameters to the puppet process. As noted in the Puppet master main page, any valid configuration parameter from puppet.conf can be specified as an argument here. Only the options that affect where Puppet will look for files should be specified here. Once puppet knows where to find puppet.conf, adding arguments here could be confusing.
With this configuration in place, we are ready to start Apache as our Puppet master. Simply start Apache with a service httpd start.
Tip
SELinux
Security
Enhanced Linux (SELinux) is a system for Linux that provides support for mandatory access controls (MAC). If your servers are running with SELinux enabled, great! You will need to make some policy changes to allow Puppet to work within passenger. The easiest way to build up your policy is to use audit2allow, which is provided in policycoreutils-python. Rotate the audit logs to get a clean log file, and then start a Puppet run. After the Puppet run, get audit2allow to build a policy module for you and insert it. Then turn SELinux back on. Refer to https://bugzilla.redhat.com/show_bug.cgi?id=1051461 for more information.
If necessary, repeat the process until everything runs cleanly. semodule will sometimes suggest enabling the allow_ypbind Boolean; this is a very bad idea. The allow_ypbind Boolean allows so many things that it is almost as bad as turning SELinux off.
Now that Puppet is running, you'll need to open the local firewall (iptables) on port 8140 to allow your nodes to connect. Then you'll need an example site.pp to get started. For testing we will create a basic site.pp that defines a default node with a single class attached to the default node as shown in the following code:
You can start a practice node or two and run their agent against the Puppet server either using --server puppet.example.com or editing the agents puppet.conf file to point at your server. Agents will by default look for an unqualified host called Puppet. Then search based on your DNS configuration (search in /etc/resolv.conf), and if you do not control DNS, you may have to edit the local /etc/hosts file to specify the IP address of your Puppet master. A sample run, for a node called node1, should look something like the following commands:
Sign the certificate on the Puppet master and run again; the run should look like the following commands:
You now have a working passenger configuration. This configuration can handle a much larger load than the default WEBrick server provided with puppet. Puppet Labs suggests the WEBrick server is appropriate for small installations; in my experience that number is much less than 100 nodes, maybe even less than 50. You can tune the passenger configuration and handle a large number of nodes, but to handle a very large installation (1000s of nodes), you'll need to start splitting up the workload.
Splitting up the workload
Puppet is a web service. But there are several different components supporting that web service, as shown in the following diagram:
Each of the different components in your Puppet infrastructure: SSL CA, Reporting, Storeconfigs, and Catalog compilation can be split up into their own server or servers.
Unless you are having issues with certificate signing consuming too many resources, it's simpler to keep the signing machine a single instance, possibly with a hot spare. Having multiple certificate signing machines means that you have to keep certificate revocation lists synchronized.
Reporting should be done on a single instance if possible. Reporting options will be shown in Chapter 7, Reporting and Orchestration.
Storeconfigs should be run on a single server, storeconfigs allows for exported resources and is optional. The recommended configuration for storeconfigs is puppetdb, which can handle several thousand nodes in a single installation.
Catalog compilation is the one task that can really bog down your Puppet installation. Splitting compilation among a pool of workers is the biggest win for scaling your deployment. The idea here is to have a primary point of contact for all your nodes—the Puppet master. Then, using proxying techniques, the master will direct requests to specific worker machines within your Puppet infrastructure. From the perspective of the nodes checking into the Puppet master, all the interaction appears to come from the main proxy machine.
To understand how we are going to achieve this load balancing, we first need to look at how the agents request data from our Puppet master. The request URL sent to our Puppet master has the format https://puppetserver:8140/environment/resource/key. The "environment" in the request URL is the Puppet environment in use by the node. It defaults to production but can be other values as we will see in later chapters. The resource being requested can be any of the accepted REST API calls, such as: catalog, certificate, resource, report, file_metadata, or
file_content. A complete listing of the http_api is available at http://docs.puppetlabs.com/guides/rest_api.html.
Requests from nodes to the Puppet masters follow a pattern that we can use to configure our proxy machine. The pattern is as follows:
For example, when node1.example.com requests its catalog in the production environment, it connects to the server and requests the following (using URL encoding):
https://puppet.example.com:8140/production/catalog/node1.example.com.
Knowing that there is a pattern to the requests, we can configure Apache to redirect requests based on regular expression matches to different machines in our Puppet infrastructure.
Our first step in splitting up our load will be to clone our Puppet master server twice to create two new worker machines, which we will call worker1.example.com and worker2.example.com. In this example, we will use 192.168.100.101 for worker1 and 192.168.100.102 for worker2. Create a private network for all the Puppet communication on 192.168.100.0/24. Our Puppet master will use the address 192.168.100.100. It is important to create a private network for the worker machines as our proxy configuration removes the SSL encryption, which means that communication between the workers and the master proxy machine is unencrypted.
Our new Puppet infrastructure is shown in the following diagram:
On our Puppet server, we will change the Apache puppet.conf as follows. Instead of listening on 8140, we will listen on 18140, and importantly, only listen on our private network as this traffic will be unencrypted. Next, we will not enable SSL on 18140. And finally we will remove any header settings we were making in our original file as shown in the following configuration:
The configuration for this VirtualHost is much simpler. Now, on the worker machines, create /etc/httpd/conf.d/puppet.conf files that are identical to the previous files but have different Listen directives shown as follows:
Remember to open port 18140 on the worker machines' firewalls (iptables) and start httpd.
Returning to the Puppet master machine, create a proxy.conf file in the Apache conf.d directory (/etc/httpd/conf.d) to point at the workers. We will create two proxy pools. The first is for certificate signing, called puppetca, as shown in the following configuration:
A second proxy pool is for catalog compilation, called puppetworker, as shown in the following configuration:
Next recreate the Puppet VirtualHost listener for 8140 with the SSL and certificate information used previously, as shown in the following configuration:
Since we know that we want all certificate requests going to the puppetca balancer, we use ProxyPassMatch to match URLs that have a certificate as the second phrase following the environment as shown in the next configuration. Our regular expression searches for a single word followed by /certificate.*, and any match is sent to our puppetca balancer.
The only thing that remains is to send all noncertificate requests to our load balancing pair, worker1 and worker2, as shown in the following configuration:
At this point, we can restart Apache on the Puppet master.
Tip
SELinux
You'll need to allow Puppet to bind to port 18140 at this point since the default puppet SELinux module allows for 8140 only. You will also need to allow Apache to connect to the worker instances; there is a Boolean for that, httpd_can_network_connect.
Now, when a node connects, if it requests for a certificate, it will be redirected to the VirtualHost on port 18140 on the Puppet master. If the node requests a catalog, it will be redirected to one of the worker nodes. To convince yourself that this is the case, edit /etc/puppet/manifests/site.pp on your worker1 node and insert notify as shown in the following configuration:
Do the same on worker2 with the message Compiled on worker2, run puppet agent again on your node, and see where the catalog is being compiled using the following commands:
Tip
You may see "Compiled on worker2", which is expected.
To verify that certificates are being handled properly, clean the certificate for your example node, remove it from the node, and restart the agent.
- On the master:
- On the node:
Tip
Alternatively to this configuration, you could use the puppetca setting in puppet.conf on your nodes to get clients to use a specific machine for signing requests.
Since this is an enterprise installation, we should have a dashboard of some kind running to collect reports from workers.
Tip
If your reports setting on the master is either HTTP or puppetdb, then this section won't affect you.
We'll clone our worker again to make a new server called reports (192.168.100.103), which will collect our reports. We then have to add another line to our Apache proxy.conf configuration file to use the new server, and we need to place this line directly after the certificate proxy line. Since reports must all be sent to the same machine to be useful, we won't use a balancer line as before, and we will simply set the proxy to the address of the reports machine directly.
Keep the /etc/httpd/conf.d/proxy.conf balancer section updated to send reports to 192.168.100.103.
Again, restart Apache and make sure that report=true is set on the node in the [agent] section of puppet.conf. Run Puppet agent on the node, and verify that the report gets sent to 192.168.100.103 (look in /var/lib/puppet/reports/).
Tip
If you are still seeing problems with client catalog compilation timeouts after creating multiple catalog workers, it may be that your client is timing out the connection before the worker has a chance to compile the catalog. Try experimenting with the configtimeout parameter in the [agent] section of puppet.conf
Setting this higher may resolve your issue. You will need to change the ProxyTimeout directive in the proxy.conf configuration for Apache as well. This will be revisited in Chapter 10, Troubleshooting.
Keeping the code consistent
At this point, we are able to scale out our catalog compilation to as many servers as we need, but we've neglected one important thing: we need to make sure that the Puppet code on all the workers remains in sync. There are a few ways we can do this, and when we cover integration with Git in Chapter 3, Git and Environments, we will see how to use Git to distribute the code.
A simple way to distribute the code is with rsync; this isn't the best solution, but just for example, you will need to run rsync whenever you change the code. This will require changing the Puppet user's shell from /sbin/nologin to /bin/bash or /bin/rbash, which is a potential security risk.
Tip
If your puppet code is on a filesystem that supports ACLs, then creating an rsync user and giving that user rights to that filesystem is a better option. Using setfacl, it is possible to grant write access to the filesystem for a user other than Puppet.
First we create an ssh-key for rsync to use to ssh between the worker nodes and the master. We then copy the key into the authorized_keys file of the Puppet user on the workers using the ssh-copy-id command as follows:
Tip
Creating SSH Keys and using rsync
The trailing slash on the first part /etc/puppet/ and the absence of the slash on the second part, puppet@worker1:/etc/puppet is by design. That way, we get the contents of /etc/puppet on the master placed into /etc/puppet on the worker.
Using rsync is not a good enterprise solution, and the concept of using SSH Keys and transferring the files as the Puppet user is the important part of this method.
A second option to keep the code consistent is to use NFS. If you already have an NAS appliance, then using the NAS to share out the Puppet code may be the simplest solution. If not, using the Puppet master as an NFS server is another, but this does make your Puppet master a big, single point of failure. NFS is not the best solution to this sort of problem.
Using a clustered filesystem such as gfs2 or glusterfs is a good way to maintain consistency between nodes. This also removes the problem of the single point of failure with NFS.
A third option is to have your version control system keep the files in sync with a post-commit hook or scripts that call Git directly, such as r10k or puppet-sync. We will cover how to configure Git to do some housekeeping for us in a later chapter. Using Git to distribute the code is a popular solution since it only updates the code when a commit is made, the continuous delivery model. If your organization would rather push code at certain points, then using the scripts mentioned earlier on a routine basis is the solution I would suggest.
Now that we have our Puppet infrastructure running on two workers and the master, you might notice that the main Apache virtual machine need not be on the same machine as the certificate-signing machine. At this point, there is no need to run passenger on that main gateway machine, and you are open to use whatever load balancing solution you see fit. In this example I will be using nginx as the main proxy point.
Tip
Using nginx is not required, but you may wish to use nginx as the proxy machine. This is because nginx has more configuration options for its proxy module, such as redirecting based on client IP address.
The important thing to remember here is that we are just providing a web service. We'll intercept the SSL part of the communication with nginx and then forward it onto our worker and CA machines as necessary. Our configuration will now look like the following diagram:
We will start with a blank machine this time; we do not need to install passenger or Puppet on the machine. To make use of the latest SSL-handling routines, we will download nginx from the nginx repository.
Now we need to copy the SSL CA files from the Puppet master to this gateway using the following commands:
Now we need to create a gateway configuration for nginx, which we will place in /etc/ngninx/conf.d/puppet-proxy.conf
We will define the two proxy pools as we did before, but using nginx syntax this time.
Next, we create a server stanza, specifying that we handle the SSL connection, and we need to set some headers before passing on the communication to our proxied servers.
Setting ssl_verify_client to optional_no_ca is important, since on the first connection, the client will not have a signed certificate, so we need to accept all connections but mark a header with the verification status.
The header X-Client-Verify will hold success or failure at this point, so our Puppet master will know if the certificate is valid. Now we need to look for certificate requests and hand those off to the puppetca pool:
Then we can send all other requests to our worker pool
Now we need to start nginx on the gateway machine, open up port 8140 on the firewall, and open up 18140 on the Puppet master firewall (gateway will now need to communicate with that port).
Running puppet again on your node will now produce the same results as before, but you are now able to leverage the load balancing of nginx over that of Apache.
Tip
You will need to synchronize the SSL CA Certificate Revocation List (CRL) from the Puppet master to the gateway machine. Without synchronization, the keys that are removed from the Puppet master will not be revoked on the gateway machine.
One last split or maybe a few more
We have already split our workload into a certificate-signing machine (the master or puppetca), a pool of catalog machines, and a report-gathering machine. What is interesting as an exercise at this point is that we can also serve files up using our gateway machine.
Based on what we know about the puppet HTTP API, we know that requests for file_buckets, and files have specific URIs that we can serve directly from nginx without using passenger or Apache or even puppet. To test the configuration, alter the definition of the example class to include a file as follows:
Create the example file in /etc/puppet/modules/example/files/example.
This file lives on the workers. On the gateway machine, rsync your Puppet module code from the workers into /var/lib/nginx/puppet. Now, to prove that the file is coming from the gateway, edit the example file after you run the rsync.
The /etc/puppet/modules/example/files/example file lives on the gateway. At this point, we can start serving up files from nginx by putting in a location clause as follows; we will do two stanzas, one for files outside modules and the other for module-provided files at /etc/nginx/conf.d/gateway.conf.
Restart nginx on the gateway machine, and then run Puppet on the node using the following command:
As we can see, although the file living on the workers has the contents "This file lives on the workers," our node is getting the file directly from nginx on the gateway.
Tip
Our node will keep changing /tmp/example to the same file each time because the catalog is compiled on the worker machine with contents different from those of the gateway. In a production environment, all the files would need to be synchronized.
One important thing to consider is security, as any configured client can retrieve files from our gateway machine. In production, you would want to add ACLs to the file location.
As we have seen, once the basic proxying is configured, further splitting up of the workload becomes a routine task. We can split the workload to scale to handle as many nodes as we require.
United States
Great Britain
India
Germany
France
Canada
Russia
Spain
Brazil
Australia
Singapore
Hungary
Philippines
Mexico
Thailand
Ukraine
Luxembourg
Estonia
Lithuania
Norway
Chile
South Korea
Ecuador
Colombia
Taiwan
Switzerland
Indonesia
Cyprus
Denmark
Finland
Poland
Malta
Czechia
New Zealand
Austria
Turkey
Sweden
Italy
Egypt
Belgium
Portugal
Slovenia
Ireland
Romania
Greece
Argentina
Malaysia
South Africa
Netherlands
Bulgaria
Latvia
Japan
Slovakia
Filter
