What do you get with Print?

Instant access to your digital eBook copy whilst your Print order is Shipped

Paperback book shipped to your preferred address

Download this book in EPUB and PDF formats

Access this title in our online reader with advanced features

DRM FREE - Read whenever, wherever and however you want

Extending SaltStack

Chapter 1. Starting with the Basics

The vast majority of Salt users see it as a configuration management platform. And in truth, it handles that very well. But it did not start off with that as a design goal. In its early days, Salt was a communication framework that was designed to be useful even to those who did not write code. But for those who were willing, it was also designed to be heavily extensible to those users who had some Python in their toolbelt.

Before we get into writing modules, it will help to have a basic understanding of how the Salt module system works. In this chapter, you'll learn the following:

How the loader system works
How Salt uses Python

Using plugins

As Salt was originally designed as a backbone that other software could use to communicate, its earliest purpose was to collect information from a large cluster of both physical and virtual machines, and return that data either to the user or to a database. Various programs, such as ps, du, and netstat, were used to collect that information. Because of that, each program was wrapped with a plugin, which contained various functions to call those programs, and parse the return data.

Those plugins were originally called modules. Later, when other types of module were added to Salt, the original modules began to be referred to as execution modules. This is because the execution modules would do the heavy lifting, and other types of module would generally wrap around them and extend their functionality.

Loading modules

Like many data centers, the one that Salt was created in had various servers that used different software packages to perform their work. One server would be running Nginx, while another would be running DNSMasq. It wouldn't make sense to enable the nginx module on the DHCP server, or a dnsmasq module on the web server. A number of popular programs solve this by allowing the user to configure which plugins will be loaded before starting the service.

Salt had a different way of handling plugins. In a large infrastructure, individual configuration of servers can be costly in terms of time. And as configuration management was added to Salt, a core belief grew that configuration management platforms should require as little configuration themselves as possible. What is the point of using such a suite to save time if so much time is required to get it going in the first place?

This is how the loader system came to be. Salt would always ship with a full set of modules, and Salt would automatically detect modules that would be available, and dynamically load them.

Execution modules are a type of plugin that performs most of the heavy lifting inside of Salt. These were the first to use the loader system, and for a short time there was no other type of module. As the functionality of Salt increased, it quickly became evident that other types of module would be needed. For instance, return output was originally just printed to the console. Then the output was changed to be easier to handle from shell scripts. Then the outputter system was added, so that output could be displayed in JSON, YAML, Python's pprint, and any other format that might be useful.

Standard modules

In the beginning, there were some types of module that would always be loaded. The first of these was the test module, which required nothing more than Salt's own dependencies; in particular, it would only require Python.

Other modules were also designed for general use, requiring no more than Salt's own dependencies. The file module would perform various file-based operations. The useradd module would wrap the standard Unix useradd program. This was fine, so long as Salt was only used on Unix-like platforms. When users started running Salt on Windows, where those utilities were not readily available, things changed. This is where virtual modules really started to shine.

Virtual modules

Supporting Salt on various platforms, such as both Unix-like and Windows, presents the same problem as whether or not to make the nginx module available: if that platform is installed and available, make the module available. Otherwise, don't. Salt handles the availability problem by implementing virtual modules.

The idea behind a virtual module is that it will contain a piece of code that will detect whether or not its dependencies are met, and if so, the module will be loaded and made available to Salt on that system. We'll get into the details of actually doing this in Chapter 2, Writing Execution Modules.

Lazy loading modules

In the beginning, if a module was detected as being loadable, then it would be loaded as the Salt service was started. A number of modules may be loaded for a particular system, which the administrator never intends to use. It may be nice to have them, but in some cases it's better to only load them when they're needed.

When the Salt service starts, the lazy loader will detect which modules may be used on a particular system, but it won't immediately load them into memory. Once a particular module is called, Salt will load it on demand, and then keep it in memory. On a system that typically only uses a small handful of modules, this can result in a much smaller footprint than before.

Extending the loader system

As we said before, the loader system was originally designed for one type of module: what we now call execution modules. Before long, other types of module were added, and that number continues to grow even today.

This book does not include every type of module, but it does cover quite a few. The following list is not comprehensive, but it will tell you much of what is available now, and possibly give you an idea of what other types of module to look at after you finish this book:

Execution modules do much of the heavy lifting inside of Salt. When a program needs to be called, an execution module will be written for it. When other modules need to use that program, they will call out to that module.
Grain modules are used to report information about Minions. Virtual modules often rely heavily on these. Configuration can also be defined in grains.
Runner modules were designed to add an element of scripting to Salt. Whereas execution modules run on Minions, a runner module would run on the Master, and call out to the Minions.
Returner modules give Minions a way to return data to something besides the Master, such as a database configured to store log data.
State modules transform Salt from a remote execution framework into a configuration management engine.
Renderer modules allow Salt States to be defined using different file formats, as appropriate.
Pillar modules extend grains, by providing a more centralized system of defining configuration.
SDB modules provide a simple database lookup. They are usually referenced from configuration areas (including grains and pillars) to keep sensitive data from appearing in plaintext.
Outputter modules affect how command-line data output is shown to the user.
External file server modules allow the files that Salt serves to be stored somewhere besides locally on the Master.
Cloud modules are used to manage virtual machines across different compute cloud providers.
Beacons allow various pieces of software, from other Salt components to third-party applications, to report data to Salt.
External authentication modules allow users to access the Master without having to have a local account on it.
Wheel modules provide an API for managing Master-side configuration files.
Proxy minion modules allow devices that cannot run the Salt platform itself to be able to be treated as if they were still full-fledged Minions.
Engines allow Salt to provide internal information and services to long-running external processes. In fact, it may be best to think of engines as programs in their own right, with a special connection to Salt.
The Master Tops system allows States to be targeted without having to use the top.sls file.
Roster modules allow Salt SSH to target Minions without having to use the /etc/salt/roster file.
Queue modules provide a means of organizing function calls.
The pkgdb and pkgfile modules allow the Salt Package Manager to store its local database and install Salt formulas into a location outside of the local hard drive.

These modules were generally created as necessity dictated. All of them are written in Python. And while some can be pretty extensive, most are pretty simple to create. In fact, a number of modules that now ship with Salt were actually provided by users who had no previous Python experience.

Loading modules with Python

Python is well suited to building a loader system. Despite being classified as a very high-level language (and not a mid-level language like C), Python has a lot of control over how it manages its own internals. The existence of robust module introspection built into Python was very useful for Salt, as it made the arbitrary loading of virtual modules at runtime a very smooth operation.

Each Salt module can support a function called __virtual__(). This is the function that detects whether or not a module will be made available to Salt on that system.

When the salt-minion service loads, it will go through each module, looking for a __virtual__() function. If none is found, then the module is assumed to have all of its requirements already met, and it can be made available. If that function is found, then it will be used to detect whether the requirements for that module are met.

If a module type uses the lazy loader, then modules that can be loaded will be set aside to be loaded when needed. Modules that do not meet the requirements will be discarded.

Detecting grains

On a Minion, the most important things to load are probably the grains. Although grain modules are important (and are discussed in Chapter 3, Extending Salt Configuration), there are in fact a number of core grains that are loaded by Salt itself.

A number of these grains describe the hardware on the system. Others describe the operating system that Salt is running on. Grains such as os and os _family are set, and used later to determine which of the core modules will be loaded.

For example, if the os_family grain is set to redhat, then the execution module located at salt/modules/yumpkg.py will be loaded as the pkg module. If the os_family grain is set to debian, then salt/modules/aptpkg.py will be loaded as the pkg module.

Using other detection methods

Grains aren't the only mechanism used for determining whether a module should be loaded. Salt also ships with a number of utilities that can be used. The salt.utils library contains a number of functions that are often faster than grains, or have more functionality than a simple name=value (also known as a key-value pair) configuration can provide.

One example is the salt.utils.is_windows() function that, as the name implies, reports whether Salt is being run inside of Windows. If Windows is detected, then salt/modules/win_file.py will be loaded as the file module. Otherwise, salt/modules/file.py will be loaded as the file module.

Another very common example is the salt.utils.which() function, which reports whether a necessary shell command is available. For instance, this is used by salt/modules/nginx.py to detect whether the nginx command is available to Salt. If so, then the nginx module will be made available.

There are a number of other examples that we could get into, but there is not nearly enough room in this book for all of them. As it is, the most common ones are best demonstrated by example. Starting with Chapter 2, Writing Execution Modules, we will begin writing Salt modules that make use of the examples that we've already gone over, plus a wealth of others.

Key benefits

Get the most up-to-date practical resource on writing new Salt modules and extending Salt

Learn through use cases and encounter both commonly-used modules as well as advanced ones

Effectively troubleshoot problems and hiccups encountered while building and putting modules to work

Description

Salt already ships with a very powerful set of tools, but that doesn't mean that they all suit your needs perfectly. By adding your own modules and enhancing existing ones, you can bring the functionality that you need to increase your productivity. Extending SaltStack follows a tutorial-based approach to explain different types of modules, from fundamentals to complete and full-functioning modules. Starting with the Loader system that drives Salt, this book will guide you through the most common types of modules. First you will learn how to write execution modules. Then you will extend the configuration using the grain, pillar, and SDB modules. Next up will be state modules and then the renderers that can be used with them. This will be followed with returner and output modules, which increase your options to manage return data. After that, there will be modules for external file servers, clouds, beacons, and finally external authentication and wheel modules to manage the master. With this guide in hand, you will be prepared to create, troubleshoot, and manage the most common types of Salt modules and take your infrastructure to new heights!

What you will learn

Understand the working of Salt s Loader system

Write several of the most common types of Salt modules

Interact between different kinds of modules and build new ones

Submit open source modules upstream to the Salt project

Make Salt interact with third-party services and applications

What do you get with Print?

Instant access to your digital eBook copy whilst your Print order is Shipped

Paperback book shipped to your preferred address

Download this book in EPUB and PDF formats

Access this title in our online reader with advanced features

DRM FREE - Read whenever, wherever and however you want

Frequently bought together

$38.99

$48.99

$54.99

Total $ 142.97

Filter reviews by

All

Amazon verified reviews

Mike S Jul 08, 2016

I was provided a free copy of this book in exchange for a truthful and honest review. I am a SaltStack Certified engineer and have been working with Salt for about 2 years. I’m a sys admin by trade, not a programmer, and haven't previously tried writing my own Salt modules beyond the simple “hello world” type tutorial in the Salt documentation. That said, this book provides excellent guidance on how to approach creating your own modules of nearly all types. As a relative Python novice, at no point was I confused about conventions or what the example code was doing. This will be a very useful reference as I continue to develop our configuration management infrastructure and further bend Salt to my will and needs. Every environment is different and having the ability to add functionality to your tools is important.The book has a few minor rough edges where examples appear to have been revised and are obviously inconsistent (reference to sshd vs mysqld and vice versa in one of the first examples) but it doesn't detract from the point. There are also some stray pylint references that I assume are leftover from a proofreading process. Overall, this is a solid reference that speaks from experience, provides relevant examples, warns of pitfalls, and provides troubleshooting tips for when you get started on your own.

Amazon Verified review

Innocent Bystander Aug 07, 2016

Extending SaltStack by Joseph Hall is a fantastic resource if you have gotten your feet wet with SaltStack and would like to do more complex management, and extend the power of SaltStack to fully automate management of your infrastructure and manage the more complex use cases possible with SaltStack.Building on is previous work which will give you a strong basis in the overall platform, SaltStack covers the more complex use cases and capabilities that can help you fully realize the extensible power of this flexible management platform. From building custom modules, to using runners to move data or using beacons to build a custom monitoring system, Hall ties the various functions of Salt together and explains very well how they all work together to build the exact functionality you need.The book is intended for people with moderate to advanced knowledge of the Salt System but also does a good job of reminding you of the basics where necessary to illustrate how a particular function works or should work to accomplish your end goal.If you are looking to increase your knowledge of the Salt system or if you are looking to build a customized, automated management and automation platform, this book is perfect, and will serve as both a teacher and a reference as you move forward. The examples for each section provide real world code and use cases to help you learn the system's intricacies and master the Salt Management platform.As someone who's worked with Salt for the last several years, I was asked to review this book and provided a copy of the book for the review.

Megan Wilhite Aug 11, 2016

This book really helped me to understand some of the more complicated aspects of saltstack that the documentation only covers on the surface. The exercises and examples were great cases to help me to understand how to write modules in salt and the book was approached with someone has very little python experience which was very useful.The book focuses on writing modules and helping the user to understand how to write modules and troubleshoot them when it is not working. This has helped me to further understand salt and to troubleshoot problems within salt by diving into the code. All around great book which is a great way to learn how to write modules and understand more of the complicated aspects of salt.

Shane Lee Jul 21, 2016

I found this book to be an excellent resource for writing your own custom modules and extensions for Salt. This book explains how Salt functions under the hood. It provides clear, concise examples and code snippets that demonstrate how to setup your environment and write Salt code.The book begins by explaining that Salt is so much more than just a configuration management platform. Then it goes through many of Salt's functionalities and demonstrates how to custom extensions for each one. It provides tips on how to approach writing custom extensions. It explains the reasoning behind design decisions that affect the way you write your code.This book covers everything you need to write custom extensions for Salt. I have worked with Salt for over a year. This book made me realize I still have a lot to learn.

Janne Aug 06, 2016

One of the key points about Salt is it being much more than Puppet, Chef or Ansible. Any book written about Salt should - in my opinion - express this clearly and make the reader understand the possibilities Salt has. I think Joseph does this well in Extending SaltStack. It is not purely a book written by a developer for developers, but contains vast amount of good information for less coding users as well, like myself.To be really efficient with Salt in the long run, you either need to dive deep in to the actual codebase, navigate through the sometimes complex/vague documentation or read books like this one. For me personally, I learned a lot even about the most basic features like runners, beacons (which I have not yet used at all), debugging and the mind of salt-cloud (which I know Joseph has contributed a lot to).What I would've liked to see more of are practical and more complex examples of Salt usecases, although it might not be the subject of this book. But maybe the book could've been about writing your own modules which come together in the end. Maybe in the next book!

Extending SaltStack: Build and write salt modules

What do you get with Print?

Extending SaltStack

Chapter 1. Starting with the Basics

Using plugins

Loading modules

Standard modules

Virtual modules

Lazy loading modules

Extending the loader system

Loading modules with Python

Detecting grains

Using other detection methods

Summary

Page 1 of 5

Key benefits

Description

Who is this book for?

What you will learn

Product Details

What do you get with Print?

Product Details

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

Filter reviews by

People who bought this also bought

About the author

FAQs

Extending SaltStack: Build and write salt modules

What do you get with Print?

Loading modules

Standard modules

Virtual modules

Lazy loading modules

Key benefits

Description

Who is this book for?

What you will learn

Product Details

What do you get with Print?

Product Details

Packt Subscriptions

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

Filter reviews by

People who bought this also bought

About the author

FAQs