The Consul system – servers and clients

Sometimes, in order to understand exactly how something works, we need to understand the components involved. After all, within every form of communication, there is a transmitter and a receiver, and if we aren't familiar with either one it can be difficult to understand the message. Consul, at its core, is a communications engine. It provides a new structure to facilitate the communications of network services, not just network devices. If you've spent any time in the networking field, you'll have heard terms such as IP address, subnet mask, and gateway address. These are all critical components involved in the communications of your network devices. However, in the glorious world of the cloud, microservices, and dynamic adjustments to the network, these components will change as the network waxes and wanes. Throughout this painful process, we've learned that the addressing system we've grown to know and love was simply a means to an end. That end is the connection and efficient communication of our services. So how does Consul do that? We are about to find out!

Consul servers – the master of their domain

Within almost any operational system or structure, there is a brain. This might not be the Brain who, along with his sidekick Pinky, attempts to take over the world day after day. We're focusing here on the brain that makes the decisions. The brain that determines whether to have Indian food or Thai food for dinner. The brain that figures out what to wear on a particular day. The brain that decides which employee gets the biggest raise. Now, at this point, you might be saying computers and applications don't have brains; they only do what we tell them to do. You are absolutely correct, but that doesn't mean that the machines don't have the logic and intelligence, even if artificial, to make decisions. In some ways, they are superior, as machines and applications are able to focus on logic and less on ego and emotion. Right, Mr. Spock? That is correct, you subservient and primitive mammal.

When contemplating the brain of Consul, we must look at the Consul server cluster. Looking at synonyms for server, we find servant, and that is exactly what the Consul server does for the distributed Consul clients (which we'll discuss later). The servers also work together, with an elected server making primary decisions, and distributing those decisions to the other servers within the overall cluster. So, what kind of decisions do these servants of my network make?

For starters, one of the most useful decisions the Consul server makes is where to find services. Everything that Consul does is predicated on the need to not only discover the services on the network, and their location, but also to share that information with any entitled entity asking for that information. If you think of the old telephone switchboards, somebody (an application) would pick up a phone and instantly be connected with a central operator (the server). The caller (application) would request a particular number (service) to be connected to, such as Transylvania 6-5000. The operator (server) would then move those giant plugs around and connect the caller (application) with the destination party (service).

Figure 1.1 – Telephone switchboard circa 1945

But what if I have applications that I don't want to be able to find specific services? Well, I'm glad you asked! In some cases, you have people or machines wishing to contact other people to talk to them about extending the warranty on their car. Perhaps I don't want to receive those messages. Well, I would tell that central operator to not allow those callers to contact me, kind of like a prehistoric Do Not Call Registry. These decisions are also managed by the Consul servers in accordance with your directive.

OK, it sounds like Consul can make a lot of cool and intelligent decisions automatically, but where does it get the information required to make those decisions? Well, how is any decision made? An employee's raise might be based upon their work performance considering they were also writing a book. What clothing we decide to wear is based on occasion and environment. What to eat for dinner can be based on…well, too many variables to list here! Every decision we make is only as good as the data we have. The Consul client provides that data.

Important Note

Within the official Consul documentation, there is a single agent that can be configured in both client and server mode. Throughout this text, we will refer to the server agents simply as the server, and the client agent simply as the client.

The cluster constituents

As the brain is the decision-maker in any system, there must be entities that will feed that data and perform the work based on that decision. For example, when the manager makes their decisions, the data is likely fed by board members, senior management executives, and hopefully team members. That being said, we all know the team members are the ones that do all of the work!

The plethora of data fed into the decisions made by the Consul server structure consists of two categories:

• Consul configuration
• Network data

The configuration of course is determined by those that are operating the Consul system, hopefully well thought out, defined as code, and peer-reviewed. The data, however, is fed by Consul clients that are distributed throughout the network. These clients may co-reside with the application, or they may travel alongside the application watching its every move. Regardless of the location, the client is not only a client for the server, but it is also a client for the application, representing the application and reporting information about the application to the server. This enables Consul's deployment within the network without the need to modify existing applications or services. Furthermore, a single client can represent multiple services and applications.

Figure 1.2 – Agent deployment options

Important Note

Care should be taken when utilizing an external service monitor with the Consul client. The data available to the client is reduced in this configuration and it is less flexible with respect to migrating services.

The Consul client not only monitors the applications that it is associated with, but it also collects information about the distance between itself and other clients within the network. This allows the servers to make more intelligent decisions about which instances of the applications or services are best suited to serve the impending request. The concept is very similar to emergency services when you dial for police or the fire service. There is one centralized number that you dial into (that is, 911 in the United States), but that centralized number connects you with whatever dispatch is closest to your current location. As Voice over IP services spread, this was a difficult challenge that increased network intelligence helped us overcome.

The marriage of server and client

OK, now that we know what the servers do, and we know what clients do, how does this entire system work together? As I've mentioned before, Consul is all about communication. The Consul servers need to communicate with each other, and the clients all need to communicate with the server cluster.

Figure 1.3 – Consul servers and agents living in harmony

The server communication is pretty manageable. It's a small dinner party with 3, 5, or 7 individuals. However, as you scale from tens to hundreds to potentially thousands of clients, having all of these clients report back to the relatively small server cluster can be quite overwhelming. Our intimate dinner party has expanded to a much larger party with tens, hundreds, or potentially thousands of guests. Can you imagine all of them wanting to talk to the head table at once! However, if the guests were able to chat amongst themselves, they could share their own information, discover new things, and only update the head table with pertinent and relevant information. In some cases, we might call this chat gossiping. Oddly enough, that's exactly the protocol Consul uses for this interaction, but we'll dig deeper into that in a later chapter.

Alright, now we have an understanding of the purpose of the servers, the purpose of the clients, and how they interact and communicate with each other. But what the heck can we do with this amazingly beautiful system? Why do I even care?