Data provenance and integrity

When working with any kind of data, two often-overlooked questions should be considered:

• Does the data come from a reliable source?
• Does the data contain all of the contents that you expected it to?

The first question relates to the topic of data provenance. Data provenance is about determining the origin of data and determining where the data originated from.

The second question refers to the topic of data integrity. Data integrity is about determining whether the contents you received from a remote location represents what you expected to receive. It helps determine whether data was tampered with as it was sent across the wire.

Both data provenance and data integrity can be verified using a concept called digital signatures. An author can create a unique signature based on cryptography to sign data, and the consumer of that data can use cryptographic tools to verify the authenticity of that signature. If the authenticity is verified, then the consumer is assured that the data originates from the expected source and was not tampered with as it was transferred.

Authors can create a digital signature using a variety of tools. One such method is by using Pretty Good Privacy (PGP). PGP, in this context, refers to OpenPGP, which is a set of standards based on encryption. PGP focuses on establishing asymmetric encryption, which is based on the use of two different keys – private and public.

Private keys are meant to be kept secret, while public keys are designed to be shared. In PGP, the private key is used to encrypt data, while the public key is used by consumers to decrypt that data. The PGP key pair is often created using a tool called GPG, which is an open source tool that implements the OpenPGP standard.

To begin working with PGP, the first step is to create the key pair, which will generate a set of public and private keys. Once the PGP key pair has been created, the author can use GPG to sign the data. When data has been signed, GPG performs the following steps in the background:

1. A hash is calculated based on the contents of the data. The output is a fixed-length string called the message digest.
2. The message digest is encrypted using the author’s private key. The output is the digital signature.

To verify the signature, consumers must use the author’s public key to decrypt it. This verification can also be performed using GPG.

Digital signatures play a role in Helm in two ways:

• First, each Helm binary has an accompanying digital signature that’s owned by one of the Helm maintainers. This signature can be used to verify the origin of the download, as well as its integrity.
• Second, Helm charts can also be digitally signed so that they benefit from the same form of verifications. Authors of Helm charts can sign the chart during packaging, and the chart users can verify the chart’s authenticity by using the author’s public key.

Now that you understand how data provenance and integrity come into play concerning digital signatures, in the next section, you will create a GPG key pair on your local machine that will be used to elaborate on the previously described concepts.

Creating a GPG key pair

To create a key pair, you must have GPG installed on your local machine. Use the following instructions as a guide to installing GPG on your respective machine. Note that this chapter is based on gpg version 2.3.6:

• For Windows, you can use the Chocolatey package manager:

  > choco install gnupg

You can also download the installer for Windows from https://gpg4win.org/download.html.

• For macOS, you can use the Homebrew package manager by using the following command:

  $ brew install gpg

You can also download the macOS-based installer from https://sourceforge.net/p/gpgosx/docu/Download/.

• For Debian-based Linux distributions, you can use the apt package manager:

  $ sudo apt install gnupg

• For RPM-based Linux distributions, you can use the dnf package manager:

  $ sudo dnf install gnupg

Once you have installed GPG, you can create your own GPG key pair, which we will use throughout our discussion on data provenance and integrity.

Follow these steps to configure your key pair:

1. First, we need to begin the generation process by running the gpg --full-generate-key command:

   $ gpg --full-generate-key

2. For the Please select what kind of key you want prompt, select (1) RSA and RSA:

   Please select what kind of key you want:
      (1) RSA and RSA
      (2) DSA and Elgamal
      (3) DSA (sign only)
      (4) RSA (sign only)
      (9) ECC (sign and encrypt) *default*
     (10) ECC (sign only)
     (14) Existing key from card
   Your selection? 1

The reason we are using RSA instead of the default option (ECC) is that ECC is not supported by the crypto library used in Helm’s source code.

3. Next, you will be prompted to enter the key size. For this example, we can simply select the default, so continue by pressing the Enter key:

   RSA keys may be between 1024 and 4096 bits long.
   What keysize do you want? (3072) <enter>
   Requested keysize is 3072 bits

4. After you enter your key size, you will be asked how long the key should be valid. Since this key will be used solely to run through the examples, we recommend setting a short expiration, such as 1 week (1w):

   Please specify how long the key should be valid.
            0 = key does not expire
         <n>  = key expires in n days
         <n>w = key expires in n weeks
         <n>m = key expires in n months
         <n>y = key expires in n years
   Key is valid for? (0) 1w
   Key expires at Sun May 22 12:26:09 2022 EDT
   Is this correct? (y/N) y

5. Now, you will be prompted for your name and email address. These will be used to identify you as the owner of the key pair and will be the name and email address displayed by those who receive your public key. You will also be prompted to provide a comment, which you can simply leave blank:

   GnuPG needs to construct a user ID to identify your key.
   Real name: John Doe
   Email address: [e-mail]
   Comment: <enter>
   You selected this USER-ID:
       "John Doe <[e-mail]>"

6. Press the O key to continue.
7. Finally, you will be prompted to enter your private key passphrase. Providing a strong passphrase is essential for protecting your identity in the event your private key is stolen. This is because it must be provided each time you attempt to access your key.

To keep our example simple, we will create an empty string passphrase to avoid passphrase prompts. While this is acceptable in this demonstration, you should protect any private key you intend to use in a real-world situation with a strong passphrase.

To continue, simply press Enter to submit an empty passphrase. When prompted, select <Yes, protection is not needed>.

Once your GPG key pair has been created, you will see an output similar to the following:

pub   rsa3072 2022-05-15 [SC] [expires: 2022-05-22]
      D2557B1EDD57BBC41A5D4DA7161DADB1C5AC21B5
uid                      John Doe <[e-mail]>
sub   rsa3072 2022-05-15 [E] [expires: 2022-05-22]

The preceding output displays information about the public (pub) and private (sub) keys, as well as the fingerprint of the public key (the second line of the output). The fingerprint is a unique identifier that’s used to identify you as the owner of that key. The third line, beginning with uid, displays the name and email address that you entered when generating the GPG key pair.

With your key pair now created, let’s continue to the next section to learn how a Helm binary can be verified.

Verifying Helm downloads

As discussed in Chapter 2, Preparing a Kubernetes and Helm Environment, one of the ways Helm can be installed is by downloading an archive from GitHub. These archives can be installed from Helm’s GitHub releases page (https://github.com/helm/helm/releases) by selecting one of the links shown in the following screenshot:

Figure 12.1 – The Installation and Upgrading section of Helm’s GitHub releases page

At the bottom of the preceding screenshot, you will notice a paragraph explaining that the release was signed. Each Helm release is signed by a Helm maintainer and can be verified against the digital signature that corresponds to the downloaded release. Each of the signatures is located under the Assets section, as shown here:

Figure 12.2 – The Assets section of Helm’s GitHub releases page

To verify the provenance and integrity of a Helm download, in addition to the binary itself, you should also download the corresponding .asc file. Note that sha256 files are used to verify the integrity only. In this example, we will download the .tar.gz.asc file, which verifies both provenance and integrity.

Let’s demonstrate how a Helm release can be verified. First, we should download a Helm archive, along with its corresponding .asc file:

1. Download a Helm archive that corresponds with your operating system. For this example, we will use version 3.8.2. If you are running an AMD64-based Linux system, the version for this distribution can be downloaded from the GitHub release page or by using the following curl command:

   $ curl -LO https://get.helm.sh/helm-v3.8.2-linux-amd64.tar.gz

2. Next, download the.asc file that corresponds with your operating system. When running an AMD64-based Linux system, helm-v3.8.2-linux-amd64.tar.gz.asc would be the resulting file that would be downloaded. You can download this file from the GitHub release page or by using the following curl command:

   $ curl -LO https://github.com/helm/helm/releases/download/v3.8.2/helm-v3.8.2-linux-amd64.tar.gz.asc

Once both files have been downloaded, you should see the two files located within the same directory on the command line:

$ ls –l
helm-v3.8.2-linux-amd64.tar.gz
helm-v3.8.2-linux-amd64.tar.gz.asc

The next step involves importing the Helm maintainer’s public key to your local GPG keyring. This allows you to decrypt the digital signature contained in the .asc file to verify the provenance and integrity of your downloaded binary. GPG public keys are saved in public key servers such as keyserver.ubuntu.com and pgp.mit.edu. As such, we can use the gpg --recv-key command to download the maintainer’s key from a public key server.

Let’s import the maintainer’s public key and continue with the verification process:

1. First, recall the maintainer’s public key fingerprint from Figure 12.1:

   672C657BE06B4B30969C4A57461449C25E36B98E

2. Use the gpg --recv-key command to download and import the key into your local keychain:

   $ gpg --recv-key 672C657BE06B4B30969C4A57461449C25E36B98E
   gpg: key 461449C25E36B98E: public key "Matthew Farina <[e-mail]>" imported
   gpg: Total number processed: 1
   gpg:               imported: 1

3. Now that the public key has been imported, you can verify the Helm release by using the --verify subcommand of GPG. This command has the gpg --verify <signature> <data> syntax:

   $ gpg --verify helm-v3.8.2-linux-amd64.tar.gz.asc helm-v3.8.2-linux-amd64.tar.gz

This command decrypts the digital signature contained in the .asc file. If it is successful, it means that the Helm download (the file ending in .tar.gz) was signed by the person you expected (Matt Farina for this release) and that the download was not modified or altered in any way. A successful output looks similar to the following:

gpg: Signature made Wed Apr 13 14:00:32 2022 EDT
gpg:                using RSA key 711F28D510E1E0BCBD5F6BFE9436E80BFBA46909
gpg: Good signature from "Matthew Farina <[e-mail]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 672C 657B E06B 4B30 969C  4A57 4614 49C2 5E36 B98E
     Subkey fingerprint: 711F 28D5 10E1 E0BC BD5F  6BFE 9436 E80B FBA4 6909

Upon further inspection of this output, you may notice the WARNING message, indicating that the key was not certified, which may lead you to question whether or not the verification was successful. In this case, the verification was indeed successful, but you have not certified the maintainer’s key yet, so GPG returns this warning.

The This key is not certified message is normally not an issue, but if you would like to ensure this warning does not appear in the future, you can follow these steps to certify the maintainer’s public key:

1. Check that the public key’s fingerprint (also referred to as the primary key from the gpg –verify output) matches the fingerprint displayed on the Helm releases page. As you will recall from Figure 12.1, the fingerprint was displayed, as shown here:

   This release was signed with 672C 657B E06B 4B30 969C 4A57 4614 49C2 5E36 B98E and can be found at @mattfarina keybase account.

2. Because the key we imported matches the fingerprint displayed on GitHub, we know that we can trust this key. Trust can be associated with this key by using the gpg --sign-key subcommand:

   $ gpg --sign-key 672C657BE06B4B30969C4A57461449C25E36B98E

In the Really sign? prompt, enter y.

Now that you have signed the maintainer’s public key, the key has been certified. The next time you perform a verification with this key, you should no longer see the warning message:

$ gpg --verify helm-v3.8.2-linux-amd64.tar.gz.asc helm-v3.8.2-linux-amd64.tar.gz
gpg: assuming signed data in 'helm-v3.8.2-linux-amd64.tar.gz'
gpg: Signature made Wed Apr 13 14:00:32 2022 EDT
gpg:                using RSA key 711F28D510E1E0BCBD5F6BFE9436E80BFBA46909
gpg: Good signature from "Matthew Farina <[e-mail]>" [full]

Digital signatures also play a role in signing and verifying Helm charts. We will explore this topic in the next section.

Signing and verifying Helm charts

Similar to how the Helm maintainers sign releases, you can sign your Helm charts so that users can verify its origin as well as confirm it contains the expected content. To sign a chart, you must have a GPG key pair present on your local workstation (we created a GPG key pair earlier in the Creating a GPG key pair section).

There is one important caveat to note before we start signing charts. If you are using GPG version 2 or greater, you must export your public and secret keyrings to a legacy format. Early versions of GPG stored keyrings in the .gpg file format, which is the format that Helm expects your keyring to be in (at the time of writing). Newer versions of GPG store keyrings in the .kbx file format, which is not compatible at the time of writing.

Luckily, we can export our keys in the .gpg format by following these steps:

1. First, find your GPG version by running the following command:

   $ gpg --version
   gpg (GnuPG) 2.3.6
   libgcrypt 1.10.1
   Copyright (C) 2021 Free Software Foundation, Inc.

2. If your GPG version is 2 or greater, export your public and secret keyrings using the gpg --export and gpg --export-secret-keys commands:

   $ gpg --export > ~/.gnupg/pubring.gpg
   $ gpg --export-secret-keys > ~/.gnupg/secring.gpg

Once your keyrings have been exported, you will be able to sign your charts using the helm package command. The helm package command provides three key flags that allow you to sign and package charts:

• --sign: This allows you to sign a chart using a PGP private key.
• --key: The name of the key to use when signing.
• --keyring: The location of the keyring containing the PGP private key.

Let’s run the helm package command to sign the guestbook Helm chart from the Packt repository:

$ helm package --sign --key <key_name> --keyring ~/.gnupg/secring.gpg helm-charts/charts/guestbook

The <key_name> placeholder refers to either the email, name, or fingerprint associated with the desired key. These details can be found by using the gpg --list-keys command.

If the helm package command is successful, you will see the following files displayed in the current directory:

guestbook-0.1.0.tgz
guestbook-0.1.0.tgz.prov

The guestbook-0.1.0.tgz file is the archive that contains the Helm chart. This file is always created by helm package, whether you are signing the chart or not.

The guestbook-0.1.0.tgz.prov file is called a provenance file. The provenance file contains a provenance record, which contains the following:

• The chart metadata from the file
• The sha256 hash of the guestbook-0.1.0.tgz file
• The PGP digital signature

Helm chart consumers leverage provenance files to verify the data provenance and integrity of the chart that they have downloaded. So, chart developers should be sure to publish both the .tgz archive as well as the .tgz.prov provenance file to their Helm chart repository.

While you have successfully signed the guestbook chart and have created the .tgz.prov file, it is not quite enough for users to verify the chart, as they still need to access your public key to decrypt your signature. You can make this key available for users by publishing it to the PGP key servers with the gpg --send-key command:

$ gpg --send-key <key_name>

End users can then download and import this key by using the gpg --recv-key command:

$ gpg --recv-key <key_name>

Once a user has imported your public key (and exported it to the ~/.gnupg/pubring.gpg keyring, as shown earlier in this section), they can verify your Helm chart by using the helm verify command, provided both the .tgz chart archive and .tgz.prov provenance file have been downloaded to the same directory:

$ helm verify --keyring ~/.gnupg/pubring.gpg guestbook-0.1.0.tgz
Signed by: John Doe <[e-mail]>
Using Key With Fingerprint: D2557B1EDD57BBC41A5D4DA7161DADB1C5AC21B5
Chart Hash Verified: sha256:c8089c7748bb0c8102894a8d70e641010b90abe9bb45962a 53468eacfbaf6731

If verification is successful, you will see that the signer, the signer’s public key, and the chart have been verified. Otherwise, an error will be returned. The verification could fail for a variety of reasons, including the following:

• The .tgz and .tgz.prov files are not in the same directory.
• The .tgz or .tgz.prov files are corrupt.
• The file hashes do not match, indicating a loss of integrity.
• The public key used to decrypt the signature does not match the private key originally used to encrypt it.

The helm verify command is designed to be run on locally downloaded charts, so users may find it better to leverage the helm install --verify command instead, which performs the verification and installation in a single command, assuming that the .tgz and .tgz.prov files can both be downloaded from a chart repository.

The following command describes how the helm install --verify command can be used:

$ helm install guestbook <chart_repo>/guestbook --verify --keyring ~/.gnupg/pubring.gpg

By using the methodologies described in this section, chart developers and consumers can be assured that the content is sourced from a trusted origin and has been unaltered.

With an understanding of how data provenance and integrity play a role in Helm, let’s continue discussing Helm security considerations by moving on to our next topic – security concerning Helm charts and Helm chart development.