OpenStack Cloud Computing Cookbook: Over 100 practical recipes to help you build and operate OpenStack cloud computing, storage, networking, and automation , Fourth Edition

The controllers (also referred to as infrastructure or infra nodes) run the heart of the OpenStack services and are the only servers exposed (via load balanced pools) to your end users. The controllers run the API services, such as Nova API, Keystone API, and Neutron API, as well as the core supporting services such as MariaDB for the database required to run OpenStack, and RabbitMQ for messaging. It is this reason why, in a production setting, these servers are set up as highly available as required. This means that these are deployed as clusters behind (highly available) load balancers, starting with a minimum of 3 in the cluster. Using odd numbers starting from 3 allows clusters to lose a single server without and affecting service and still remain with quorum (minimum numbers of votes needed). This means that when the unhealthy server comes back online, the data can be replicated from the remaining 2 servers (which are, between them, consistent), thus ensuring data consistency.

Networking is recommended to be highly resilient, so ensure that Linux has been configured to bond or aggregate the network interfaces so that in the event of a faulty switch port, or broken cable, your services remain available. An example networking configuration for Ubuntu can be found in Appendix A.

These are the servers that run the hypervisor or container service that OpenStack schedules workloads to when a user requests a Nova resource (such as a virtual machine). These are not too dissimilar to hosts running a hypervisor, such as ESXi or Hyper-V, and OpenStack Compute servers can be configured in a very similar way, optionally using shared storage. However, most installations of OpenStack forgo the need for the use of shared storage in the architecture. This small detail of not using shared storage, which implies the virtual machines run from the hard disks of the compute host itself, can have a large impact on the users of your OpenStack environment when it comes to discussing the resiliency of the applications in that environment. An environment set up like this pushes most of the responsibility for application uptime to developers, which gives the greatest flexibility of a long-term cloud strategy. When an application relies on the underlying infrastructure to be 100% available, the gravity imposed by the infrastructure ties applications to specific data center technology to keep it running. However, OpenStack can be configured to introduce shared storage such as Ceph (http://ceph.com/) to allow for operational features such as live-migration (the ability to move running instances from one hypervisor to another with no downtime), allowing enterprise users move their applications to a cloud environment in a very safe way. These concepts will be discussed in more detail in later chapters on compute and storage. As such, the reference architecture for a compute node is to expect virtual machines to run locally on the hard drives in the server itself.

With regard to networking, like the controllers, the network must also be configured to be highly available. A compute node that has no network available might be very secure, but it would be equally useless to a cloud environment! Configure bonded interfaces in the same way as the controllers. Further information for configuring bonded interfaces under Ubuntu can be found in Appendix A.

Storage in OpenStack refers to block storage and object storage. Block storage (providing LUNs or hard drives to virtual machines) is provided by the Cinder service, while object storage (API driven object or blobs of storage) is provided by Swift or Ceph. Swift and Ceph manage each individual drive in a server designated as an object storage node, very much like a RAID card manages individual drives in a typical server. Each drive is an independent entity that Swift or Ceph uses to write data to. For example, if a storage node has 24 x 2.5in SAS disks in, Swift or Ceph will be configured to write to any one of those 24 disks. Cinder, however, can use a multitude of backends to store data. For example, Cinder can be configured to communicate with third-party vendors such as NetApp or Solidfire arrays, or it can be configured to talk to Sheepdog or Ceph, as well as the simplest of services such as LVM. In fact, OpenStack can be configured in such a way that Cinder uses multiple backends so that a user is able to choose the storage applicable to the service they require. This gives great flexibility to both end users and operators as it means workloads can be targeted at specific backends suitable for that workload or storage requirement.

This book briefly covers Ceph as the backend storage engine for Cinder. Ceph is a very popular, highly available open source storage service. Ceph has its own disk requirements to give the best performance. Each of the Ceph storage nodes in the preceding diagram are referred to as Ceph OSDs (Ceph Object Storage Daemons). We recommend starting with 5 of these nodes, although this is not a hard and fast rule. Performance tuning of Ceph is beyond the scope of this book, but at a minimum, we would highly recommend having SSDs for Ceph journaling and either SSD or SAS drives for the OSDs (the physical storage units).

The differences between a Swift node and a Ceph node in this architecture are very minimal. Both require an interface (bonded for resilience) for replication of data in the storage cluster, as well as an interface (bonded for resilience) used for data reads and writes from the client or service consuming the storage.

The primary difference is the recommendation to use SSDs (or NVMe) as the journaling disks.

The end users of the OpenStack environment expect services to be highly available, and OpenStack provides REST API services to all of its features. This makes the REST API services very suitable for placing behind a load balancer. In most deployments, load balancers would usually be highly available hardware appliances such as F5. For the purpose of this book, we will be using HAProxy. The premise behind this is the same though—to ensure that the services are available so your end users can continue working in the event of a failed controller node.

Operating installing System: Ubuntu 16.04 x86_64

We assume that each server has Ubuntu 16.04 installed.

Log in, as root, onto each server that will have OpenStack installed.

Configuration of the host's networking, on a Ubuntu system, is performed by editing the /etc/network/interfaces file.

We have configured the physical networking of our hosts to ensure a good known state and configuration for running OpenStack. Each of the interfaces configured here is specific to OpenStack—either directly managed by OpenStack (for example, br-vlan) or used for inter-service communication (for example, br-mgmt). In the former case, OpenStack utilizes the br-vlan bridge and configures tagged interfaces on bond1 directly.

Note that the convention used here, of VLAN tag ID using a portion of the subnet, is only to highlight a separation of VLANs to specific subnets (for example, bond0.236 is used by the 172.29.236.0/24 subnet). This VLAN tag ID is arbitrary, but must be set up in accordance with your specific networking requirements.

Finally, we performed a fairly rudimentary test of the network. This gives you the confidence that the network configuration that will be used throughout the life of your OpenStack cloud is fit for purpose and gives assurances in the event of a failure of a cable or network card.

Ensure that you are root on the deployment host. In most cases, this is the first infrastructure controller node that we have named for the purposes of this book to be called infra01. We will be assuming that all Ansible commands will be run from this host, and that it expects to be able to connect to the rest of the servers on this network over the host network via SSH.

In order to allow a hands-free, orchestrated OpenStack-Ansible deployment, follow these steps to create and propagate root SSH public key of infra01 across all servers required of the installation:

We first generated a key pair file for use by SSH. The -t option specified the rsa type encryption, -f specified the output of the private key, where the public portion will get .pub appended to its name, and -N "" specified that no passphrase is to be used on this key. Consult your own security standards if the presented options differ from your company's requirements.

Ensure that you are root on the deployment host. In most cases, this is the first infrastructure controller node, infra01.

At this point, we will be checking out the version of OpenStack-Ansible from GitHub.

To set up Ansible and its dependencies, follow these steps:

The OpenStack-Ansible project provides a handy script to ensure that Ansible and the right dependencies are installed on the deployment host. This script (bootstrap-ansible.sh) lives in the scripts/ directory of the checked out OpenStack-Ansible code, so at this stage we need to grab the version we want to deploy using Git. Once we have the code, we can execute the script and wait for it to complete.

Visit https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest for more information.

Ensure that you are root on the deployment host. In most cases, this is the first infrastructure controller node, infra01.

Let's assume that you're using the first infrastructure node, infra01, as the deployment server.

If you have not followed the preceding Installing Ansible, playbooks, and dependencies recipe review, then as root, carry out the following if necessary:

git clone -b 16.05 https://github.com/openstack/openstack-ansible.git /opt/openstack-ansible

This downloads the OpenStack-Ansible playbooks to the /opt/openstack-ansible directory.

To configure our OpenStack deployment, carry out the following steps:

Ansible is a very popular server configuration tool that is well-suited to the task of installing OpenStack. Ansible takes a set of configuration files that Playbooks (a defined set of steps that get executed on the servers) use to control how they executed. For OpenStack-Ansible, configuration is split into two areas: describing the physical environment and describing how OpenStack is configured.

The first configuration file, /etc/openstack_deploy/openstack_user_config.yml, describes the physical environment. Each section is described here:

cidr_networks:
  container: 172.29.236.0/24
  tunnel: 172.29.240.0/24
  storage: 172.29.244.0/24

This section describes the networks required for an installation based on OpenStack-Ansible. Look at the following diagram to see the different networks and subnets.

The used_ips: section refers to IP addresses that are already in use on that subnet, or reserved for use by static devices. Such devices are load balancers or other hosts that are part of the subnets that OpenStack-Ansible would otherwise have randomly allocated to containers:

global_overrides:
  internal_lb_vip_address: 172.29.236.117
  external_lb_vip_address: 192.168.1.117
  tunnel_bridge: "br-vxlan"
  management_bridge: "br-mgmt"
  storage_bridge: "br-storage"

The global_overrides: section describes the details around how the containers and bridged networking are set up. OpenStack-Ansible's default documentation expects Linux Bridge to be used; however, Open vSwitch can also be used. Refer to Chapter 4, Neutron – OpenStack Networking, for more details.

The internal_lb_vip_address: and external_lb_vip_address: sections refer to the private and public sides of a typical load balancer. The private (internal_lb_vip_address) is used by the services within OpenStack (for example, Nova calls communicating with the Neutron API would use internal_lb_vip_address, whereas a user communicating with the OpenStack environment once it has been installed would use external_lb_vip_address). See the following diagram:

A number of load balance pools will be created for a given Virtual IP (VIP) address, describing the IP addresses and ports associated with a particular service, and for each pool—one will be created on the public/external network (in the example, a VIP address of 192.168.100.117 has been created for this purpose), and another VIP for use internally by OpenStack (in the preceding example, the VIP address 172.29.236.117 has been created for this purpose).

The tunnel_bridge: section is the name given to the bridge that is used for attaching the physical interface that participates in the VXLAN tunnel network.

The management_bridge: section is the name given to the bridge that is used for all of the OpenStack services that get installed on the container network shown in the diagram.

The storage_bridge: section is the name given to the bridge that is used for traffic associated with attaching storage to instances or where Swift proxied traffic would flow.

Each of the preceding bridges must match the names you have configured in the /etc/network/interfaces file on each of your servers.

The next section, provider_networks, remains relatively static and untouched as it describes the relationship between container networking and the physical environment. Do not adjust this section.

Following the provider_networks section are the sections describing which server or group of servers run a particular service. Each block has the following syntax:

service_name:
  ansible_inventory_name_for_server:
    IP_ADDRESS
  ansible_inventory_name_for_server:
    IP_ADDRESS

A number of sections and their use are listed here:

Ensure that you are the root user on the deployment host. In most cases, this is the first infrastructure controller node, infra01.

To install OpenStack using the OpenStack-Ansible playbooks, you navigate to the playbooks directory of the checked out Git repository, then execute each playbook in turn:

Installation of OpenStack using OpenStack-Ansible is conducted using a number of playbooks. The first playbook, setup-hosts.yml, sets up the hosts by laying down the container configurations. At this stage, Ansible knows where it will be placing all future services associated with OpenStack, so we use the dynamic inventory information to perform an installation of HAProxy and configure it for all the services used by OpenStack (that are yet to be installed). The next playbook, setup-infrastructure.yml, configures and installs the base Infrastructure services containers that OpenStack expects to be present, such as Galera. The final playbook is the main event—the playbook that installs all the required OpenStack services we specified in the configuration. This runs for quite a while—but at the end of the run, you are left with an installation of OpenStack.

The OpenStack-Ansible project provides a wrapper script to the ansible command that would ordinarily run to execute Playbooks. This is called openstack-ansible. In essence, this ensures that the correct inventory and configuration information is passed to the ansible command to ensure correct running of the OpenStack-Ansible playbooks.

We will step through a problem that caused one of the playbooks to fail.

Note the failed playbook and then invoke it again with the following steps:

Should there be a failure at this first stage, execute the following:

In some situations, it might be applicable to destroy the installation and begin again. As each service gets installed in LXC containers, it is very easy to wipe an installation and start from the beginning. To do so, carry out the following steps:

Ansible is not perfect and so are computers. Sometimes failures occur in the environment due to SSH timeouts, or some other transient failure. Also, despite Ansible trying its best to retry the execution of a playbook, the result might be a failure. Failure in Ansible is quite obvious—it is usually predicated by outputs of red text on the screen. In most cases, rerunning the offending playbook may get over some transient problems. Each playbook runs a specific task, and Ansible will state which task has failed. Troubleshooting why that particular task had failed will eventually lead to a good outcome. Worst case, you can reset your installation from the beginning.

Ensure that you are root on the first infrastructure controller node, infra01.

The installation of OpenStack-Ansible creates several utility containers on each of the infra nodes. These utility hosts provide all the command-line tools needed to try out OpenStack, using the command line of course. Carry out the following steps to get access to a utility host and run various commands in order to verify an installation of OpenStack manually:

The OpenStack-Ansible method of installing OpenStack installs OpenStack services into isolated containers on our Linux servers. On each of the controller (or infra) nodes are about 12 containers, each running a single service such as nova-api or RabbitMQ. You can view the running containers by logging into any of the servers as root and issuing a lxc-ls -f command. The -f parameter gives you a full listing showing the status of the instance such as whether it is running or stopped.

One of the containers on the infra nodes has utility in its name, and this is known as a utility container in OpenStack-Ansible terminology. This container has OpenStack client tools installed, which makes it a great place to start manually testing an installation of OpenStack. Each container has at least an IP address on the container network—in the example used in this chapter this is the 172.29.236.0/24 subnet. You can SSH to the IP address of this container, or use another lxc command to attach to it: lxc-attach -n <name_of_container>. Once you have a session inside the container, you can use it like any other system, provided those tools are available to the restricted four-walls of the container. To use OpenStack commands, however, you first need to source the resource environment file which is named openrc. This is a normal bash environment file that has been prepopulated during the installation and provides all the required credentials needed to use OpenStack straight away.

Ensure that you are root on the deployment host. In most cases, this is the first infrastructure controller node, infra01.

The following are the common changes and how they can be changed using Ansible. As we'll adjust the configuration, all of these commands are executed from the same host you used to perform the installation.

To adjust the CPU overcommit/allocation ratio, carry out the following steps:

For more complex changes, for example, to add configuration that isn't a simple one-line change in a template, we can use an alternative in the form of overrides. To make changes to the default Nova Quotas, carry out the following as an example:

Changes for Neutron, Glance, Cinder, and all other services are modified in a similar way. Adjust the name of the service in the syntax used. For example, to change a configuration item in the neutron.conf file, you would use the following syntax:

neutron_neutron_conf_overrides:  
  DEFAULT:
   dhcp_lease_duration = -1

Then execute the following commands:

cd /opt/openstack-ansible/playbooks
openstack-ansible os-neutron-install.yml --tag 'neutron-config'

We modified the same OpenStack-Ansible configuration files as in the Configuring the installation recipe and executed the openstack-ansible playbook command, specifying the playbook that corresponded to the service we wanted to change. As we were making configuration changes, we notified Ansible of this through the --tag parameter.

Refer to https://docs.openstack.org/ for all configuration options for each service.

In order to run a multi-node OpenStack environment, running as a virtual environment on your laptop or designated host, the following set of requirements are needed:

To run the OpenStack environment within the virtual environment, we need a few programs installed, all of which are free to download and use: VirtualBox, Vagrant, and Git. VirtualBox provides the virtual servers representing the servers in a normal OpenStack installation; Vagrant describes the installation in a fully orchestrated way; Git allows us to check out all of the scripts that we provide as part of the book to easily test a virtual OpenStack installation. The following instructions describe an installation of these tools on Ubuntu Linux.

We first need to install VirtualBox if it is not already installed. We recommend downloading the latest available releases of the software. To do so on Ubuntu Linux as root, follow these steps:

Once VirtualBox is installed, we can install Vagrant. Follow these steps to install Vagrant:

The lab utilizes two vagrant plugins: vagrant-hostmanager and vagrant-triggers. To install these, carry out the following steps:

If Git is not currently installed, issue the following command to install git on a Ubuntu machine:

apt update
apt install git

Now that we have the required tools, we can use the OpenStackCookbook Vagrant lab environment to perform a fully orchestrated installation of OpenStack in a VirtualBox environment:

Vagrant is an awesome tool for orchestrating many different virtual and cloud environments. It allows us to describe what virtual servers need to be created, and using Vagrant's provisioner allows us to run scripts once a virtual machine has been created.

Vagrant's environment file is called Vagrantfile. You can edit this file to adjust the settings of the virtual machine, for example, to increase the RAM or number of available CPUs.

This allows us to describe a complete OpenStack environment using one command:

vagrant up

The environment consists of the following:

Once the environment has finished installing, you can use the environment by navigating to http://192.168.100.10/ in your web browser. To retrieve the admin password, follow the steps given here and view the file named openrc.

There is a single controller node that has a utility container configured for use in this environment. Attach to this with the following commands:

vagrant ssh controller-01
sudo -i
lxc-attach -n (lxc-ls | grep utility)
openrc

Once you have retrieved the openrc details, copy these to your openstack-client virtual machine. From here you can operate OpenStack, mimicking a desktop machine accessing an installation of OpenStack utilizing the command line.

vagrant ssh openstack-client
openrc

You should now be able to use OpenStack CLI tools to operate the environment.