Setting up the Kubernetes cluster on Linux via Ansible (kubespray)

If you are familiar with configuration management, such as Puppet, Chef and Ansible, kubespray (https://github.com/kubernetes-incubator/kubespray) is the best choice to set up a Kubernetes cluster from scratch. It provides the Ansible playbook that supports the majority of Linux distributions and public clouds, such as AWS and GCP.

Ansible (https://www.ansible.com) is a Python-based SSH automation tool that can configure Linux as your desired state based on the configuration, which is called playbook. This cookbook describes how to use kubespray to set up Kubernetes on Linux.

Getting ready

As of May 2018, the latest version of kubespray is 2.5.0, which supports the following operation systems to install Kubernetes:

• RHEL/CentOS 7
• Ubuntu 16.04 LTS

According to the kubespray documentation, it also supports CoreOS and debian distributions. However, those distributions may need some additional steps or have technical difficulties. This cookbook uses CentOS 7 and Ubuntu 16.04 LTS.

In addition, you need to install Ansible on your machine. Ansible works on Python 2.6, 2.7, and 3.5 or higher. macOS and Linux might be the best choice to install Ansible because Python is preinstalled by most of macOS and Linux distributions by default. In order to check which version of Python you have, open a Terminal and type the following command:

//Use capital V
$ python -V
Python 2.7.5

Overall, you need at least three machines, as mentioned in the following table:

Type of host	Recommended OS/Distribution
Ansible	macOS or any Linux which has Python 2.6, 2.7, or 3.5
Kubernetes master	RHEL/CentOS 7 or Ubuntu 16.04 LTS
Kubernetes node	RHEL/CentOS 7 or Ubuntu 16.04 LTS

There are some network communicating with each other, so you need to at least open a network port (for example, AWS Security Group or GCP Firewall rule) as:

• TCP/22 (ssh): Ansible to Kubernetes master/node host
• TCP/6443 (Kubernetes API server): Kubernetes node to master
• Protocol 4 (IP encapsulated in IP): Kubernetes master and node to each other by Calico

In Protocol 4 (IP encapsulated in IP), if you are using AWS, set an ingress rule to specify aws ec2 authorize-security-group-ingress --group-id <your SG ID> --cidr <network CIDR> --protocol 4. In addition, if you are using GCP, set the firewall rule to specify as cloud compute firewall-rules create allow-calico --allow 4 --network <your network name> --source-ranges <network CIDR>.

Installing pip

The easiest way to install Ansible, is to use pip, the Python package manager. Some of newer versions of Python have pip already (Python 2.7.9 or later and Python 3.4 or later):

1. To confirm whether pip is installed or not, similar to the Python command, use -V:

//use capital V
$ pip -V
pip 9.0.1 from /Library/Python/2.7/site-packages (python 2.7)

2. On the other hand, if you see the following result, you need to install pip:

//this result shows you don't have pip yet
$ pip -V
-bash: pip: command not found

3. In order to install pip, download get-pip.py and install by using the following command:

//download pip install script
$ curl -LO https://bootstrap.pypa.io/get-pip.py


//run get-pip.py by privileged user (sudo)
$ sudo python get-pip.py 
Collecting pip
  Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)
    100% |################################| 1.3MB 779kB/s 
Collecting wheel
  Downloading wheel-0.30.0-py2.py3-none-any.whl (49kB)
    100% |################################| 51kB 1.5MB/s 
Installing collected packages: pip, wheel
Successfully installed pip-9.0.1 wheel-0.30.0


//now you have pip command
$ pip -V
pip 9.0.1 from /usr/lib/python2.7/site-packages (python 2.7)

Installing Ansible

Perform the following steps to install Ansible:

1. Once you have installed pip, you can install Ansible with the following command:

//ran by privileged user (sudo)
$ sudo pip install ansible

pip scans your Python and installs the necessary libraries for Ansible, so it may take a few minutes to complete.

2. Once you have successfully installed Ansible by pip, you can verify it with the following command and see output as this:

$ which ansible
/usr/bin/ansible

$ ansible --version
ansible 2.4.1.0

Installing python-netaddr

Next, according to kubespray's documentation (https://github.com/kubernetes-incubator/kubespray#requirements), it needs the python-netaddr package. This package can also be installed by pip, as shown in the following code:

$ sudo pip install netaddr

Setting up ssh public key authentication

One more thing, as mentioned previously, Ansible is actually the ssh automation tool. If you log on to host via ssh, you have to have an appropriate credential (user/password or ssh public key) to the target machines. In this case, the target machines mean the Kubernetes master and nodes.

Due to security reasons, especially in the public cloud, Kubernetes uses only the ssh public key authentication instead of ID/password authentication.

To follow the best practice, let's copy the ssh public key from your Ansible machine to the Kubernetes master/node machines:

If you've already set up ssh public key authentication between the Ansible machine to Kubernetes candidate machines, you can skip this step.

1. In order to create an ssh public/private key pair from your Ansible machine, type the following command:

//with –q means, quiet output
$ ssh-keygen -q

2. It will ask you to set a passphrase. You may set or skip (empty) this, but you have to remember it.
3. Once you have successfully created a key pair, you can see the private key as ~/.ssh/id_rsa and public key as ~/.ssh/id_rsa.pub. You need to append the public key to the target machine under ~/.ssh/authorized_keys, as shown in the following screenshot:

4. You need to copy and paste your public key to all Kubernetes master and node candidate machines.
5. To make sure your ssh public key authentication works, just ssh from the Ansible machine to the target host that won't ask for your logon password, as here:

//use ssh-agent to remember your private key and passphrase (if you set)
ansible_machine$ ssh-agent bash
ansible_machine$ ssh-add
Enter passphrase for /home/saito/.ssh/id_rsa: Identity added: /home/saito/.ssh/id_rsa (/home/saito/.ssh/id_rsa)


//logon from ansible machine to k8s machine which you copied public key
ansible_machine$ ssh 10.128.0.2
Last login: Sun Nov  5 17:05:32 2017 from 133.172.188.35.bc.googleusercontent.com
k8s-master-1$

Now you are all set! Let's set up Kubernetes using kubespray (Ansible) from scratch.

How to do it...

kubespray is provided through the GitHub repository (https://github.com/kubernetes-incubator/kubespray/tags), as shown in the following screenshot:

Because kubespray is an Ansible playbook, not a binary, you can download the latest version (as of May 2018, version 2.5.0 is the latest) of the zip or tar.gz to your Ansible machine directly and unarchive it with the following command:

//download tar.gz format
ansible_machine$ curl -LO https://github.com/kubernetes-incubator/kubespray/archive/v2.5.0.tar.gz


//untar
ansible_machine$ tar zxvf v2.5.0.tar.gz 


//it unarchives under kubespray-2.5.0 directory
ansible_machine$ ls -F
get-pip.py  kubespray-2.5.0/  v2.5.0.tar.gz


//change to kubespray-2.5.0 directory
ansible_machine$ cd kubespray-2.5.0/

Maintaining the Ansible inventory

In order to perform the Ansible playbook, you need to maintain your own inventory file, which contains target machine IP addresses:

1. There is a sample inventory file under the inventory directory, so you can copy it by using the following:

//copy sample to mycluster
ansible_machine$ cp -rfp inventory/sample inventory/mycluster

//edit hosts.ini
ansible_machine$ vi inventory/mycluster/hosts.ini 

2. In this cookbook, we are using target machines that have the following IP addresses:
   • Kubernetes master : 10.128.0.2
   • Kubernetes node : 10.128.0.4
3. In this case, hosts.ini should be in the following format:

4. Please change the IP address to match your environment.

Note that hostname (my-master-1 and my-node-1) will be set by the kubespray playbook based on this hosts.ini, so feel free to assign a meaningful hostname.

Running the Ansible ad hoc command to test your environment

Before running the kubespray playbook, let's check whether hosts.ini and Ansible itself work properly or not:

1. To do that, use the Ansible ad hoc command, using the ping module, as shown in the following screenshot:

2. This result indicates SUCCESS. But if you see the following error, probably the IP address is wrong or the target machine is down, so please the check target machine first:

3. Next, check your authority whether you can escalate a privilege on the target machine or not. In other words, whether you can run sudo or not. This is because you will need to install Kubernetes, Docker, and some related binaries, and configurations that need a root privilege. To confirm that, add the -b (become) option, as shown in the following screenshot:

4. With the -b option, it actually tries to perform sudo on the target machine. If you see SUCCESS, you are all set! Go to the How it works… section to run kubespray.

If you're unfortunate enough to see some errors, please refer to the following section to solve Ansible issues.

Ansible troubleshooting

The ideal situation would be to use the same Linux distribution, version, settings, and logon user. However, the environment will be different based on policy, compatibility, and other reasons. Ansible is flexible and can support many use cases to run ssh and sudo.

Need to specify a sudo password

Based on your Linux machine setting, you may see the following error when adding the -b option. In this case, you need to type your password while running the sudo command:

In this case, add -K (ask for the sudo password) and run again. It will ask for your sudo password when running the Ansible command, as shown in the following screenshot:

If your Linux uses the su command instead of sudo, adding --become-method=su to run the Ansible command could help. Please read the Ansible documentation for more details : http://docs.ansible.com/ansible/latest/become.html

Need to specify different ssh logon user

Sometimes you may need to ssh to target machines using a different logon user. In this case, you can append the ansible_user parameter to an individual host in hosts.ini. For example:

• Use the username kirito to ssh to my-master-1
• Use the username asuna to ssh to my-node-1

In this case, change hosts.ini, as shown in the following code:

my-master-1 ansible_ssh_host=10.128.0.2 ansible_user=kirito
my-node-1 ansible_ssh_host=10.128.0.4 ansible_user=asuna

Need to change ssh port

Another scenario is where you may need to run the ssh daemon on some specific port number rather than the default port number 22. Ansible also supports this scenario and uses the ansible_port parameter to the individual host in hosts.ini, as shown in the following code (in the example, the ssh daemon is running at 10022 on my-node-1):

my-master-1 ansible_ssh_host=10.128.0.2
my-node-1 ansible_ssh_host=10.128.0.4 ansible_port=10022

Common ansible issue

Ansible is flexible enough to support any other situations. If you need any specific parameters to customize the ssh logon for the target host, read the Ansible inventory documentation to find a specific parameter: http://docs.ansible.com/ansible/latest/intro_inventory.html

In addition, Ansible has a configuration file, ansible.cfg, on top of the kubespray directory. It defines common settings for Ansible. For example, if you are using a very long username that usually causes an Ansible error, change ansible.cfg to set control_path to solve the issue, as shown in the following code:

[ssh_connection]
control_path = %(directory)s/%%h-%%r

If you plan to set up more than 10 nodes, you may need to increase ssh simultaneous sessions. In this case, adding the forks parameter also requires you to increase the ssh timeout from 10 seconds to 30 seconds by adding the timeout parameter, as shown in the following code:

[ssh_connection]
forks = 50
timeout = 30

The following screenshot contains all of the preceding configurations in ansible.cfg:

For more details, please visit the Ansible configuration documentation at http://docs.ansible.com/ansible/latest/intro_configuration.html

How it works...

Now you can start to run the kubepray playbook:

1. You've already created an inventory file as inventory/mycluster/hosts.ini. Other than hosts.ini, you need to check and update global variable configuration files at inventory/mycluster/group_vars/all.yml.
2. There are a lot of variables defined, but at least one variable, bootstrap_os , needs to be changed from none to your target Linux machine. If you are using RHEL/CentOS7, set bootstrap_os as centos. If you are using Ubuntu 16.04 LTS, set bootstrap_os as ubuntu as shown in the following screenshot:

You can also update other variables, such as kube_version, to change or install a Kubernetes version. For more details, read the documentation at https://github.com/kubernetes-incubator/kubespray/blob/master/docs/vars.md.

3. Finally, you can execute the playbook. Use the ansible-playbook command instead of the Ansible command. Ansible-playbook runs multiple Ansible modules based on tasks and roles that are defined in the playbook.
4. To run the kubespray playbook, type the ansible-playbook command with the following parameters:

//use –b (become), -i (inventory) and specify cluster.yml as playbook
$ ansible-playbook -b -i inventory/mycluster/hosts.ini cluster.yml 

The ansible-playbook argument parameter is the same as the Ansible command. So, if you need to use -K (ask for the sudo password) or --become-method=su, you need to specify for ansible-playbook as well.

5. It takes around 5 to 10 minutes to complete based on the machine spec and network bandwidth. But eventually you can see PLAY RECAP, as shown in the following screenshot, to see whether it has succeeded or not:

6. If you see failed=0 like in the preceding screenshot, you have been successful in setting up a Kubernetes cluster. You can ssh to the Kubernetes master machine and run the /usr/local/bin/kubectl command to see the status, as shown in the following screenshot:

7. The preceding screenshot shows that you have been successful in setting up the Kubernetes version 1.10.2 master and node. You can continue to use the kubectl command to configure you Kubernetes cluster in the following chapters.
8. Unfortunately, if you see a failed count of more than 0, the Kubernetes cluster has probably not been set up correctly. Because failure is caused by many reasons, there is no single solution. It is recommended that you append the verbose option -v to see more detailed output from Ansible, as shown in the following code:

//use –b (become), -i (inventory) and –v (verbose)
$ ansible-playbook -v -b -i inventory/mycluster/hosts.ini cluster.yml

9. If the failure is timeout, just retrying the ansible-playbook command again may solve it. Because Ansible is designed as an idempotency, if you re-perform the ansible-playbook command twice or more, Ansible still can configure correctly.

10. If the failure is change target IP address after you run ansible-playbook (for example, re-using the Ansible machine to set up another Kubernetes cluster), you need to clean up the fact cache file. It is located under /tmp directory, so you just delete this file, as shown in the following screenshot:

See also

This section describes how to set up the Kubernetes cluster on the Linux OS using kubespray. It is the Ansible playbook that supports major Linux distribution. Ansible is simple, but due to supporting any situation and environment, you need to care about some different use cases. Especially with ssh and sudo-related configurations, you need to understand Ansible deeper to fit it with your environment.