In this article by Rihards Olups, the author of the book Zabbix Network Monitoring, Second Edition, we will learn how communication between Zabbix components is done in plaintext by default. In many environments, this is not a significant problem, but monitoring over the Internet in plaintext is likely not a good approach. In previous Zabbix versions, there was no built-in solution, and various VPN, stunnel, and SSH port forwarding solutions were used. Such solutions can still be used, but version 3.0 is the first Zabbix version to provide built-in encryption. In this article, we will set up several of the components to use different methods of encryption.

(For more resources related to this topic, see here.)

Overview

For Zabbix communication encryption, two methods are supported:

Preshared Key
Certificate-based encryption

The Preshared Key (PSK) method is very easy to set up but is likely harder to scale. Certificate-based encryption can be more complicated to set up but easier to manage on a larger scale and potentially more secure.

This encryption is supported between all Zabbix components: server, proxy, agent, and even zabbix_sender and zabbix_get.

For outgoing connections (such as server-to-agent or proxy-to-server), one method may be used (no encryption, PSK or certificate-based). For incoming connections, multiple methods may be allowed. This way, an agent could work with encryption by default and then turn off encryption with zabbix_get for debugging.

Backend libraries

Behind the scenes, Zabbix encryption can use one of three different libraries: OpenSSL, GnuTLS, or mbedTLS. Which one to choose? If using packages, the easiest and safest method is to start with whichever one the packages are compiled with. If compiling from source, choose the one that is easiest to compile with. The Zabbix team has made a significant effort of implementing support for all three libraries to look as similar as possible from the user's perspective. There could be differences regarding support for some specific features, but those are likely to be more obscure ones; if such problems do come up later, switching from one library to another should be fairly easy. While in most cases it would likely not matter much which library you are using, it's a good idea to know it—one good reason for supporting these three different libraries is also the ability to switch to a different library if the currently used one has a security vulnerability.

These libraries are used in a generic manner, and you don't need to use the same library for different Zabbix components—it's totally fine to use one library on the Zabbix server, another on the Zabbix proxy, and yet another with zabbix_sender.

In this article, we will try out encryption with the Zabbix server and zabbix_sender first, then move on to encrypting agent traffic using both PSK and certificate-based encryption. If you have installed from the packages, your server most likely already supports encryption. Verify this by looking at the server and agent startup messages:

TLS support:         YES

If you compiled from source and there is no TLS support, recompile the server and agent adding one of these parameters: --with-openssl, --with-gnutls, or --with-mbedtls.

Preshared key encryption

Let's start with a simple situation: a single new host that will accept PSK-encrypted connections only, to which we will send some values using zabbix_sender. For this to work, both the Zabbix server and zabbix_sender must be compiled with Transport Layer Security (TLS) support. The PSK configuration consists of a PSK identity and key. The identity is a string that is not considered to be secret—it is not encrypted during the communication, so do not put sensitive information in the identity string. The key is a hexadecimal string.

Zabbix requires the key it to be at least 32 characters long. The maximum in Zabbix is 512 characters, but it might depend on the specific version of the backend library you are using.

We could just type the key in manually, but a slightly easier method is using the openssl command:

$ openssl rand -hex 64

This will generate a 512-byte key, which we will use in a moment. Navigate to Configuration | Hosts, click on Create host, and fill in these values:

Hostname: An encrypted host
Groups: Have only Linux Servers in the In Groups block

Switch to the Encryption tab, and in the Connections from host section, leave only PSK marked. In the PSK identity field, enter secret, and paste the key we generated earlier in the PSK field:

encrypting-zabbix-traffic-img-0

When done, click on the Add button at the bottom. Take a look at the AGENT ENCRYPTION column for this host:

The first block has only one field and currently says NONE. For connections to the agent, only one method is possible; thus, this column must be showing the currently selected method for outgoing connections from the server's perspective. The second block has three fields. We can choose a combination of incoming connection methods; thus, this column must be showing which types of incoming connections from the server's perspective are accepted for this host.

Now, click on Items next to Encrypted host, and click on Create item. Fill in these values:

Name: Beers in the fridge
Type: Zabbix trapper
Key: fridge.beers

Click on the Add button at the bottom. Let's try to send a value now using following command:

$ zabbix_sender -z 127.0.0.1 -s "Encrypted host" -k fridge.beers -o 1

It should fail and show you the following message:

info from server: "processed: 0; failed: 1; total: 1; seconds spent: 0.000193"

Notice how the processed count is 0 and the failed count is 1. Let's check the Zabbix server logfile:

12254:20160122:231030.702 connection of type "unencrypted" is not allowed for host "Encrypted host" item "fridge.beers" (not every rejected item might be reported)

Now this is actually quite a helpful message—we did not specify any encryption for zabbix_sender, but we did require encrypted connection for our host. Notice the text in parentheses—if multiple items on the same host fail because of this reason, we might only see some of them, and searching the logfile by item key only might not reveal the reason.

Now is the time to get the PSK working for zabbix_sender. Run it with the --help parameter and look at the TLS connection options section. Oh yes, there's quite a lot of those. Luckily, for PSK encryption, we only need three of them: --tls-connect, --tls-psk-identity, and tls-psk-file. Before running the command, create a file called zabbix_encrypted_host_psk.txt and paste the hex key we generated earlier into it.

It is more secure to create an empty file first, change its permissions to 400 or 600, and paste the key in the file afterwards. This way, another user won't have a chance to snatch the key from the file.

Run zabbix_sender again, but with the three additional encryption parameters we just set:

$ zabbix_sender -z 127.0.0.1 -s "Encrypted host" -k fridge.beers -o 1 --tls-connect psk --tls-psk-identity secret --tls-psk-file zabbix_encrypted_host_psk.txt

With this command, we set the connection method to psk with the --tls-connect flag and specified the PSK identity and key file.

Zabbix does not support specifying the PSK key on the command line for security reasons. It must be passed in from a file.

This time, the value should be sent successfully:

info from server: "processed: 1; failed: 0; total: 1; seconds spent: 0.000070"

To be sure, verify that this item now has data in the frontend.

Certificate-based encryption

With PSK-based encryption protecting our sensitive Zabbix trapper item, let's move to the certificates. We will generate certificates for the Zabbix server and agent and require encrypted connections on the Zabbix agent side for passive items.

You might have a certificate infrastructure in your organization, but for our first test, we will generate all required certificates ourselves. We will need a new Certificate Authority (CA) that will sign our certificate. Zabbix does not support self-signed certificates.

It is highly recommended to use intermediate certificate authorities to sign client and server certificates. We will not use them in the following simple example.

Being our own authority

We'll start by creating the certificates in a separate directory. For simplicity's sake, let's do this on a test host.

The following is not intended to be a good practice. It is actually doing quite a few bad and insecure things to obtain the certificates faster. Do not follow these steps for any production setup:

$ mkdir zabbix_ca
$ chmod 700 zabbix_ca
$ cd zabbix_ca

Generate the root CA key:

$ openssl genrsa -aes256 -out zabbix_ca.key 4096

When prompted, enter a password twice to protect the key. Generate and self-sign the root certificate like this:

$ openssl req -x509 -new -key zabbix_ca.key -sha256 -days 3560 -out zabbix_ca.crt

When prompted, enter the password you used for the key before. Fill in the values as prompted—the easiest way is supplying empty values for most except the country code and common name. The common name does not have to be anything too meaningful for our test, so using a simple string such as zabbix_ca will suffice. Now, let's move on to creating a certificate we will use for the Zabbix server; first, let's generate server key and Certificate Signing Request (CSR):

$ openssl genrsa -out zabbix_server.key 2048
$ openssl req -new -key zabbix_server.key -out zabbix_server.csr

When prompted, enter the country code and the common name string as before. The common name does not have to match the server or agent name or anything else, so using a simple string such as zabbix_server will suffice. Now, let's sign this request:

$ openssl x509 -req -in zabbix_server.csr -CA zabbix_ca.crt -CAkey zabbix_ca.key -CAcreateserial -out zabbix_server.crt -days 1460 -sha256

When prompted, enter the CA passphrase. Let's continue with the certificate we will use for the Zabbix agent. Generate the agent key and certificate signing request:

$ openssl genrsa -out zabbix_agent.key 2048
$ openssl req -new -key zabbix_agent.key -out zabbix_agent.csr

Again, when prompted, enter the country code and common name string as before. The common name does not have to match the server or agent name or anything else, so using a simple string such as zabbix_agent will suffice. Let's sign the request:

$ openssl x509 -req -in zabbix_agent.csr -CA zabbix_ca.crt -CAkey zabbix_ca.key -CAcreateserial -out zabbix_agent.crt -days 1460 -sha256

When prompted, enter the CA passphrase. We're done with creating our test certificates.

Both keys were created unencrypted. Zabbix does not support prompting for the key password at this time.

Setting up Zabbix with certificates

Now, we move on to making the passive items on our test host using the certificates we just generated. We must provide the certificates to the Zabbix agent. In the directory where the Zabbix agent configuration file is located, create a new directory called zabbix_agent_certs. Restrict access to it like this:

# chown zabbix zabbix_agent_certs
# chmod 500 zabbix_agent_certs

From the directory where we generated the certificates, copy the relevant certificate files over to the new directory:

# cp zabbix_ca.crt /path/to/zabbix_agent_certs/
# cp zabbix_agent.crt /path/to/zabbix_agent_certs/
# cp zabbix_agent.key /path/to/zabbix_agent_certs/

Edit zabbix_agentd.conf and modify these parameters:

TLSAccept=cert
TLSConnect=unencrypted
TLSCAFile=/path/to/zabbix_agent_certs/zabbix_ca.crt
TLSCertFile=/path/to/zabbix_agent_certs/zabbix_agent.crt
TLSKeyFile=/path/to/zabbix_agent_certs/zabbix_agent.key

This will make the agent only accept connections when they are encrypted and use a certificate signed by that CA, either directly or through intermediates. We'll still use unencrypted connections for active items. A user could supply certificates and expect all communication to be encrypted now, which would not be the case if both the TLSAccept and TLSConnect parameters did not require it; thus, Zabbix enforces them when certificates are supplied. Restart the Zabbix agent.

Let's take a look at the host configuration list in the Zabbix frontend:

encrypting-zabbix-traffic-img-2

Looks like connections to our test host do not work anymore. Let's check the agent logfile:

2820:20160125:194427.623 failed to accept an incoming connection: from 127.0.0.1: unencrypted connections are not allowed

Looks like we broke it. We did set up encryption on the agent but did not get to configuring the server side. What if we would like to roll out encryption to all the agents and deal with the server later? In that case, it would be best to set TLSAccept=cert, unencrypted. Then, the agent would still accept unencrypted connections from our server. Once the certificates were deployed and configured on the Zabbix server, we'd only have to remove unencrypted from that parameter and restart the Zabbix agents. Let's try this out—change zabbix_agentd.conf again:

TLSAccept=cert, unencrypted

Restart the agent daemon and observe the monitoring resuming from the Zabbix server. Now let's make the server use its certificate. We'll place the certificate in a place where the Zabbix server can use it. In the directory where the Zabbix server configuration file is located, create a new directory called zabbix_server_certs. Restrict access to it using these commands:

# chown zabbix zabbix_server_certs
# chmod 500 zabbix_server_certs

If using packages that run the Zabbix server with a different username such as zabbixs or zabbixserv, replace the username in these two commands.

From the directory where we generated the certificates, copy them over to the new directory:

# cp zabbix_ca.crt /path/to/zabbix_server_certs/
# cp zabbix_server.crt /path/to/zabbix_server_certs/
# cp zabbix_server.key /path/to/zabbix_server_certs/

Edit zabbix_server.conf and modify these parameters:

TLSCAFile=/path/to/zabbix_server_certs/zabbix_ca.crt
TLSCertFile=/path/to/zabbix_server_certs/zabbix_server.crt
TLSKeyFile=/path/to/zabbix_server_certs/zabbix_server.key

Now, restart the Zabbix server. Although we have specified the certificates in both the agent and server, the passive items still work in unencrypted mode. Let's proceed with making them encrypted. In the Zabbix frontend, navigate to Configuration | Hosts, click on A test host, and switch to the Encryption tab. In the Connections to host selection, choose Certificate, then click on the Update button. After the server configuration cache is updated, it will switch to using certificate-based encryption for this host.

Going back to our scenario where we slowly rolled out certificate-based configuration to our agents and added it to the server later, we can now disable unencrypted connections on the agent side. Change this line in zabbix_agentd.conf:

TLSAccept=cert

Restart the agent. If we had followed this process from the very beginning, monitoring would have continued uninterrupted.

Let's try to use zabbix_get:

$ zabbix_get -s 127.0.0.1 -k system.cpu.load
zabbix_get [5746]: Check access restrictions in Zabbix agent configuration

That fails because agent only accepts encrypted connections now. Like we did for zabbix_sender, we can specify the certificate, but we must use the Zabbix server certificate now:

$ zabbix_get -s 127.0.0.1 -k system.cpu.load --tls-connect cert --tls-ca-file /path/to/zabbix_server_certs/zabbix_ca.crt --tls-cert-file /path/to/zabbix_server_certs/zabbix_server.crt --tls-key-file /path/to/zabbix_server_certs/zabbix_server.key

0.030000

Certainly, this results in a more secure environment. It is not enough to spoof the IP address to access this agent. It is not enough to have an account on the Zabbix server to have access to all agents—access to the server certificate is required, too. On the other hand, it makes debugging a bit more complicated.

We used PSK and certificate-based encryption with zabbix_sender and passive agent, but the same principles apply for active agents and zabbix_get. As an exercise, try to get the active agent items working with encryption too.

Concerns and further reading

At this time, encryption is a very new feature in Zabbix. While it has been developed and tested extremely carefully and pedantically, it is likely to receive further improvements. Make sure to read through the official documentation on encryption for more details and in case of changes being made. Right now, we will touch upon basic concerns and features that are missing.

In this article, we covered the Zabbix server, agent, zabbix_get and zabbix_sender—what about Zabbix proxies? Zabbix proxies fully support encryption. The configuration on the proxy side is very similar to the agent configuration, and the configuration on the server or frontend side is done in a similar way to host encryption configuration, too. Keep in mind that all involved components must be compiled with TLS support—any proxies you have might have to be recompiled. When considering encryption, think about the areas where it's needed most: maybe you have Zabbix server and proxy communicating over the Internet while all other connections are in local networks. In that case, it might make sense to set up encryption only for server-proxy communication at first. Note that encryption is not supported when communicating with the Zabbix Java gateway, but one could easily have the gateway communicate with the Zabbix proxy on the localhost, which in turn provides encryption for the channel to the Zabbix server.

We already figured out how the upgrading and transitioning to encryption can happen seamlessly without interrupting data collection—the ability of all components to accept various connection types allows us to roll the changes out sequentially.

Another reason why one might want to implement encryption only partially is performance. Currently, Zabbix does not reuse connections, implement TLS session caches, or use any other mechanism that would avoid setting up encrypted connections from scratch every time. This can be especially devastating if you have lots of passive agent items. Make sure to understand the potential impact before reconfiguring it all.

Encryption is not currently supported for authentication purposes. That is, we can not omit active agent hostnames and figure out which host it is based on the certificate alone. Similarly, we can not allow only encrypted connections for active agent auto-registration.

For certificate-based encryption, we only specified the certificates and CA information. If the CA used is large enough, it would not be very secure: any certificate signed by that CA would be accepted. Zabbix also allows verifying both the issuer and subject of the remote certificate. Unless you are using an internal CA that is used for Zabbix only, it is highly recommended you limit issuers and subjects.

Summary

In this article, we explored Zabbix's built-in encryption, which is supported between all components: the server, proxy, agent, zabbix_sender, and zabbix_get. While not supported for the Java gateway, a Zabbix proxy could easily be put in front of the gateway to provide encryption back to the Zabbix server.

Zabbix supports PSK and TLS certificate-based encryption and can use one of three different backend libraries: OpenSSL, GnuTLS, or mbedTLS. In case of security or other issues with one library, users have an option of switching to another library.

The upgrade and encryption deployment can be done in steps. All Zabbix components can accept multiple connection methods at the same time. In our example, the agent would be set up to accept both encrypted and unencrypted connections, and when we would be done with all agents, we would switch to encrypted connections on the server side. Once that would be verified to work as expected, unencrypted connections could be disabled on the agents.

With the encryption being built-in and easy to set up, it is worth remembering that encrypted connections will need more resources and that Zabbix does not support connection pooling or other methods that could decrease the load. It might be worth securing the most important channels first, leaving endpoints for later. For example, encrypting the communication between the Zabbix server and proxies would likely be a priority over connections to individual agents.