At the very start of the IT revolution, most applications were deployed directly on physical hardware, over the host OS. Because of that single user space, runtime was shared between applications. The deployment was stable, hardware-centric, and had a long maintenance cycle. It was mostly managed by an IT department, and gave much less flexibility to developers. In such cases, the hardware resources were underutilized most of the time. The following diagram depicts such a setup:

For flexible deployments, and in order to better utilize the resources of the host system, virtualization was invented. With hypervisors, such as KVM, XEN, ESX, Hyper-V, and so on, we emulated the hardware for virtual machines (VMs) and deployed a guest OS on each virtual machine. VMs can have a different OS than their host; this means that we are responsible for managing patches, security, and the performance of that VM. With virtualization, applications are isolated at VM level and are defined by the life cycle of VMs. This gives us a better return on our investment and higher flexibility at the cost of increased complexity and redundancy. The following diagram depicts a typical virtualized environment:

Since virtualization was developed, we have been moving towards more application-centric IT. We have removed the hypervisor layer to reduce hardware emulation and complexity. The applications are packaged with their runtime environment, and are deployed using containers. OpenVZ, Solaris Zones, and LXC are a few examples of container technology. Containers are less flexible compared to VMs; for example, we cannot run Microsoft Windows on Linux OS as of writing. Containers are also considered less secure than VMs, because with containers, everything runs on the host OS. If a container gets compromised, then it might be possible to get full access to the host OS. It can be a bit too complex to set up, manage, and automate. These are a few of the reasons why we have not seen the mass adoption of containers in the last few years, even though we had the technology. The following diagram shows how an application is deployed using containers:

With Docker, containers suddenly became first-class citizens. All big corporations, such as Google, Microsoft, Red Hat, IBM, and others, are now working to make containers mainstream.

Docker was started as an internal project by dotCloud founder Solomon Hykes. It was released as open source in March 2013 under the Apache 2.0 license. With dotCloud's platform as a service experience, the founders and engineers of Docker were aware of the challenges of running containers. So with Docker, they developed a standard way to manage containers.

Docker uses the operating system's underlying kernel features, which enable containerization. The following diagram depicts the Docker platform and the kernel features used by Docker. Let's look at some of the major kernel features that Docker uses:

Namespaces are the building blocks of a container. There are different types of namespace, and each one of them isolates applications from the others. They are created using the clone system call. You can also attach to existing namespaces. Some of the namespaces used by Docker will be explained in the following sections.

The PID namespace allows each container to have its own process numbering. Each PID forms its own process hierarchy. A parent namespace can see the children namespaces and affect them, but a child can neither see the parent namespace nor affect it.

If there are two levels of hierarchy, then at the top level, we would see the process running inside the child namespace with a different PID. So a process running in a child namespace would have two PIDs: one in the child namespace and the other in the parent namespace. For example, if we run a program on the container.sh container, then we can see the corresponding program on the host as well.

On the container, the sh container.sh process has a PID of 8:

On the host, the same process has a PID of 29778:

With the PID namespace, we can run the same program multiple times in different isolated environments; for example, we can run different instances of Apache on different containers. But without the net namespace, we would not be able to listen on port 80 on each one of them. The net namespace allows us to have different network interfaces on each container, which solves the problem I mentioned earlier. Loopback interfaces would be different in each container as well.

To enable networking in containers, we create pairs of special interfaces in two different net namespaces and allow them to talk to each other. One end of the special interface resides inside the container and the other resides on the host system. Generally, the interface inside the container is called eth0, and on the host system, it is given a random name, such as veth516cc56. These special interfaces are then linked through a bridge (docker0) on the host to enable communication between the containers and the route packets.

Inside the container, you will see something like the following:

$ docker container run -it alpine ash
# ip a

On the host, it would look like the following:

$ ip a

Also, each net namespace has its own routing table and firewall rules.

The inter-process communication (IPC) namespace provides semaphores, message queues, and shared memory segments. It is not widely used these days, but some programs still depend on it.

If the IPC resource created by one container is consumed by another container, then the application running on the first container could fail. With the IPC namespace, processes running in one namespace cannot access resources from another namespace.

Using only a chroot, you can inspect the relative paths of the system from a chrooted directory/namespace. The mnt namespace takes the idea of chroots to the next level. With the mnt namespace, a container can have its own set of mounted filesystems and root directories. Processes in one mnt namespace cannot see the mounted filesystems of another mnt namespace.

With the UTS namespace, we can have different hostnames for each container.

With user namespace support, we can have users who have a nonzero ID on the host, but who can have a zero ID inside the container. This is because the user namespace allows mappings of users and groups IDs per namespace.

There are ways to share namespaces between the host and container, and other containers as well. We'll see how to do this in subsequent chapters.

Control groups (cgroups) provide resource limitations and accounting for containers. The following quote is from the Linux Kernel documentation:

In simple terms, they can be compared to the ulimit shell command or the setrlimit system call. Instead of setting the resource limit to a single process, cgroups allow you to limit resources to a group of processes.

Control groups are split into different subsystems, such as CPU, CPU sets, memory block I/O, and so on. Each subsystem can be used independently, or can be grouped with others. The features that cgroups provide are as follows:

• Resource limiting: For example, one cgroup can be bound to specific CPUs, so that all processes in that group would run on given CPUs only
• Prioritization: Some groups may get a larger share of CPUs
• Accounting: You can measure the resource usage of different subsystems for billing
• Control: You can freeze and restart groups

Some of the subsystems that can be managed by cgroups are as follows:

• blkio: Sets I/O access to and from block devices, such as disks, SSDs, and so on
• Cpu: Limits access to CPU
• Cpuacct: Generates CPU resource utilization
• Cpuset: Assigns CPUs on a multicore system to tasks in a cgroup
• Devices: Grants devices access to a set of tasks in a group
• Freezer: Suspends or resumes tasks in a cgroup
• Memory: Sets limits on memory use by tasks in a cgroup

There are multiple ways to control work with cgroups. Two of the most popular ones are accessing the cgroup virtual filesystem manually and accessing it with the libcgroup library. To use libcgroup on Linux, run the following command to install the required packages on Ubuntu or Debian:

$ sudo apt-get install cgroup-tools

To install the required packages on CentOS, Fedora, or Red Hat, use the following code:

$ sudo yum install libcgroup libcgroup-tools

Once installed, you can get the list of subsystems and their mount point in the pseudo filesystem with the following command:

$ lssubsys -M

Although we haven't looked at the actual commands yet, let's assume that we are running a few containers and want to get the cgroup entries for a container. To get those, we first need to get the container ID and then use the lscgroup command to get the cgroup entries of a container, which we can get using the following command:

The union filesystem allows the files and directories of separate filesystems, known as layers, to be transparently overlaid to create a new virtual filesystem. While starting a container, Docker overlays all the layers attached to an image and creates a read-only filesystem. On top of that, Docker creates a read/write layer that is used by the container's runtime environment. You can read the Pulling an image and running a container recipe of this chapter for more details. Docker can use several union filesystem variants, including AUFS, Btrfs, zfs, overlay, overlay2, and DeviceMapper.

Docker also has a virtual file system (VFS) storage driver. A VFS doesn't support copy-on-write (COW) and is not a union filesystem. This means that each layer is a directory on the disk, and each time a new layer is created, it requires a deep copy of its parent layer. For these reasons, it has lower performance and requires more disk space, but it is a robust and stable option that works in every environment.

Docker Engine combines the namespaces, control groups, and UnionFS into a wrapper called a container format. In 2015, Docker donated its container format and runtime to an organization called the the Open Container Initiative (OCI). The OCI is a lightweight, open-governance structure formed under the Linux Foundation by Docker and other industry leaders. The purpose of the OCI is to create open industry standards around container formats and runtimes. There are currently two specifications: the Runtime Specification and the Image Specification.

The Runtime Specification outlines how to run an OCI runtime filesystem bundle. Docker donated runC, (https://github.com/opencontainers/runc) its OCI-compliant runtime to the OCI, to serve as the reference implementation.

The OCI image format contains the information needed to launch the application on the target platform. The specification defines how to create the OCI image, and what the desired output would look like. The output would consist of an image manifest, a filesystem (layer) serialization, and the image configuration. Docker donated its Docker V2 image format to the OCI to form the basis of the OCI image specification.

There are currently two container engines that support the OCI Runtime and Image Specifications: Docker and rkt.

Docker is supported on many Linux platforms, such as RHEL, Ubuntu, Fedora, CentOS, Debian, Arch Linux, among others. It is also supported on many cloud platforms, such as Amazon Web Services, Digital Ocean, Microsoft Azure, and Google Cloud. Docker has also released desktop applications for Microsoft Windows and Mac OS X that allows you to easily get Docker up and running directly on your local machine.

In this recipe, we will verify the requirements for Docker installation. We will look at a system with an Ubuntu 18.04 LTS installation, though the same steps should work on other Linux flavors as well.

Log in as a root user on the system that has Ubuntu 18.04 installed.

Perform the following steps:

1. Docker is not supported on 32-bit architecture. To check the architecture on your system, run the following command:

        $ uname -i
        x86_64

2. Docker is supported on kernel 3.8 or later. It has been back-ported on some of the kernel 2.6, such as RHEL 6.5 and above. To check the kernel version, run the following command:

      $ uname -r
      4.15.0-29-generic

3. Running the kernel should support an appropriate storage backend. Some of the options for such a backend are VFS, DeviceMapper, AUFS, Btrfs, zfs, and Overlayfs.

For Ubuntu, the default storage backend or driver is overlay2, which has been available since Ubuntu 14.04. Another popular one is DeviceMapper, which uses the device-mapper thin provisioning module to implement layers. It should be installed by default on a majority of Linux platforms. To check for device-mapper, you can run the following command:

            $ grep device-mapper /proc/devices
            253 device-mapper

On most distributions, AUFS would require a modified kernel.

4. Support for cgroups and namespaces have been in the kernel for sometime, and should be enabled by default. To check for their presence, you can look at the corresponding configuration file of the kernel you are running. For example, on Ubuntu, I can do something like the following:

    $ grep -i namespaces /boot/config-4.15.0-29-generic
    CONFIG_NAMESPACES=y

    $ grep -i cgroups /boot/config-4.15.0-29-generic
    CONFIG_CGROUPS=y

Docker requires that the host system meets a basic set of requirements in order for it to run correctly. By running the preceding commands, we were able to confirm that our system meets those requirements.

Check out the installation document on the Docker website at https://docs.docker.com/install/.

There are a few different versions of Ubuntu that are available. In this recipe, we will be installing Docker on Ubuntu 18.04, which is the latest LTS version as of writing. These same steps should also work with Ubuntu 16.04.

Check for the prerequisites mentioned in the previous recipe.

Uninstall any older versions of Docker. Previous versions of the Docker package are called docker, docker.io, or docker-engine. If these are installed, then we need to uninstall them, or else they might cause problems:

    $ sudo apt-get remove docker docker-engine docker.io

Go through the following steps:

1. Update the apt package index:

   $ sudo apt-get update

2. Install the packages to allow apt to use a repository over HTTPS:

       $ sudo apt-get install \
        apt-transport-https \
        ca-certificates \
        curl \
        software-properties-common

3. Add Docker's official GPG key:

        $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
    OK

Verify that we have the correct key installed:

        $ sudo apt-key fingerprint 0EBFCD88
      pub rsa4096 2017-02-22 [SCEA]
            9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88
      uid [ unknown] Docker Release (CE deb) <docker@docker.com>
      sub rsa4096 2017-02-22 [S]

4. Add the Docker apt repository using the stable channel:

       $ sudo add-apt-repository \
        "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
        $(lsb_release -cs) \
        stable"

5. Update the apt package index so that it includes the Docker repository we just added:

       $ sudo apt-get update

6. Install the latest version of Docker CE:

       $ sudo apt-get install docker-ce

7. Verify that the installation worked:

       $ sudo docker container run hello-world

The preceding command will install Docker on Ubuntu and all the packages required by it.

The default Docker daemon configuration file is located at /etc/docker, which is used while starting the daemon. Here are some basic operations:

• To start the service, enter the following:

    $ sudo systemctl start docker

• To verify the installation, enter the following:

    $ docker info

• To update the package, enter the following:

    $ sudo apt-get update

• To enable the start of the service at boot time, enter the following:

    $ sudo systemctl enable docker

• To stop the service, enter the following:

    $ sudo systemctl stop docker

For more information, check out the Ubuntu installation document on the Docker website at https://docs.docker.com/install/linux/docker-ce/ubuntu/.

Another popular Linux distribution is CentOS, which is a free, enterprise-class distribution that is compatible with Red Hat Enterprise Linux (RHEL). Go through the following easy recipe to install Docker on CentOS 7.x.

The centos-extra repository must be enabled. This is usually enabled by default, but if you disabled it, please enable it again.

Previously, the Docker package had a different name: It was called docker or docker-engine; it is now called docker-ce. We will need to remove any previous Docker versions in order to prevent any conflicts:

$ sudo yum remove docker \
    docker-client \
    docker-client-latest \
    docker-common \
    docker-latest \
    docker-latest-logrotate \
    docker-logrotate \
    docker-selinux \
    docker-engine-selinux \
    docker-engine

Go through the following steps:

1. Install the required packages:

      $ sudo yum install -y yum-utils \
          device-mapper-persistent-data \
          lvm2

2. Set up the Docker yum repository using the stable channel:

        $ sudo yum-config-manager \
          --add-repo \
            https://download.docker.com/linux/centos/docker-ce.repo

3. Optional: Enable the test channel for access to the nightly builds:

        $ sudo yum-config-manager --enable docker-ce-test

4. Install the latest version of docker-ce:

         $ sudo yum install docker-ce

5. If prompted to accept the GPG key, verify that it matches 060A 61C5 1B55 8A7F 742B 77AA C52F EB6B 621E 9F35. If it does, then accept it:

    Retrieving key from https://download.docker.com/linux/centos/gpg
    Importing GPG key 0x621E9F35:
     Userid : "Docker Release (CE rpm) <docker@docker.com>"
     Fingerprint: 060a 61c5 1b55 8a7f 742b 77aa c52f eb6b 621e 9f35
     From : https://download.docker.com/linux/centos/gpg
    Is this ok [y/N]: y

6. Start the Docker daemon:

        $ sudo systemctl start docker

7. Verify that the installation worked:

        $ docker container run hello-world

The preceding recipe installs Docker on CentOS and all the packages required by it.

The default Docker daemon configuration file is located at /etc/docker, which is used while starting the daemon. Here are some basic operations:

• To start the service, enter the following:

    $ sudo systemctl start docker

• To verify the installation, enter the following:

    $ docker info

• To update the package, enter the following:

    $ sudo yum -y upgrade

• To enable the service start at boot time, enter the following:

    $ sudo systemctl enable docker

• To uninstall Docker, enter the following:

        $ sudo yum remove docker-ce

• To stop the service, enter the following:

        $ sudo systemctl stop docker

For more information, check out the CentOS installation document on the Docker website at https://docs.docker.com/install/linux/docker-ce/centos/.

In the previous two recipes, we went through the different steps required to install Docker on Ubuntu and CentOS. Those steps are fine when you are only installing it on a host or two, but what if you need to install it on a hundred? In that case, you would want something a little more automated to speed up the process. This recipe shows you how to install Docker on different Linux flavors using an install script that is provided by Docker.

Like all scripts that you download off the internet, the first thing you should do is examine the script and make sure you know what it is doing before you use it. To do this, go through the following steps:

1. Visit https://get.docker.com in your favorite web browser to review the script, and make sure you are comfortable with what it is doing. If in doubt, don't use it.
2. The script needs to be run as root or with sudo privileges.
3. If Docker has already been installed on the host, it needs to be removed before running the script.

The script currently works with the following flavors of Linux: CentOS, Fedora, Debian, Ubuntu, and Raspbian.

To use the script, go through the following steps:

1. Download the script to the host system:

        $ curl -fsSL get.docker.com -o get-docker.sh

2. Run the script:

        $ sudo sh get-docker.sh

The preceding recipe used an automated script to install Docker on Linux.

In order to upgrade Docker, you will need to use the package manager on your host. Rerunning the script can cause issues if it attempts to re-add repositories that were already added. See the previous recipes to learn how to upgrade Docker on CentOS and Ubuntu using their respective package managers.

Docker for Windows is a native application that is deeply integrated with Hyper-V virtualization and the Windows networking and filesystems. It is a full-featured development environment that can be used for building, debugging, and testing Docker apps on a Windows PC. It also works well with VPNs and proxies to make it easier when used in a corporate environment.

Docker for Windows supports both Windows and Linux containers out of the box, and it is easy to switch between the two to build your multiplatform applications. It comes with the Docker CLI client, Docker Compose, Docker Machine, and Docker Notary.

Recent releases have also added Kubernetes support so that you can easily create a full Kubernetes environment on your machine with just the click of a button.

Docker for Windows has the following system requirements:

• 64-bit Windows 10 Pro, Enterprise, and Education (1607 Anniversary Update, Build 14393 or later)
• Virtualization must be enabled in BIOS and be CPU-SLAT-capable.
• 4 GB of RAM

To install Docker for Windows, go through the following steps:

1. Download Docker for Windows from the Docker Store at https://store.docker.com/editions/community/docker-ce-desktop-windows. You will need to log in in order to download the installer. If you do not have a Docker account, you can create one at https://store.docker.com/signup.
2. Double-click the installation file that you downloaded from the store. It should be called something like Docker for windows Installer.exe:

Once the installation is complete, it will automatically start up. You will notice a little whale icon in the notification area of your task bar. If you need to change any settings, right-click on the icon and select Settings.

3. Open up a command-line terminal and check to make sure that the installation is working:

            $ docker container run hello-world

This recipe will show you how to install a Docker development environment on your Windows machine.

Now that you have Docker for Windows installed, check out the following tips to get the most out of your installation:

• Docker for Windows supports both Windows and Linux containers. If you want to switch, you just need to right-click on the whale icon, select Switch to Windows containers..., and then click the Switch button:

To switch back, do the same thing, except this time, select Switch to Linux containers....

• Docker for Windows will automatically check for new updates and let you know when a new version is available to install. If you agree to upgrade, it will download the new version and install it for you.
• Kubernetes doesn't run by default. If you want to turn it on, you will need to right-click on the Docker whale icon in your task bar, then select Settings. Inside the Settings menu, there is a Kubernetes tab. Click on the tab, and then click the Enable Kubernetes option and hit the Apply button:

For more information about Docker for Windows, and for links to labs and more examples, go to https://docs.docker.com/docker-for-windows/.

Docker for Mac is the fastest and most reliable way to run Docker on a Mac. It installs all of the tools required to set up a complete Docker development environment on your Mac. It includes the Docker command line, Docker Compose, and Docker Notary. It also works well with VPNs and proxies to make it easier when used in a corporate environment.

Recent releases have also added Kubernetes support so that you can easily create a full Kubernetes environment on your machine with just the click of a button.

Docker for Mac has the following system requirements:

• macOS El Capitan 10.11, or a newer macOS release
• At least 4 GB of RAM
• The Mac hardware must be a 2010 or newer model, with Intel's hardware support for Memory Management Unit (MMU) virtualization, including Extended Page Tables (EPT) and unrestricted mode. To see whether your machine supports this, run the following command in a terminal:

        $ sysctl kern.hv_support
        kern.hv_support: 1

To install Docker for Mac, go through the following steps:

1. Download Docker for Mac from the Docker Store at https://store.docker.com/editions/community/docker-ce-desktop-mac. You will need to log in in order to download the installer. If you do not have a Docker account, you can create one at https://store.docker.com/signup.
2. Open the installation file that you downloaded from the store. It should be called something like Docker.dmg.

3. Drag and drop the whale icon into the Applications folder:

4. Double-click the Docker.app icon in the Applications folder to start Docker, as shown in the following screenshot:

5. You will be prompted to authorize Docker.app with your system password. This is normal—Docker.app needs privileged access to install some of its components. Click OK and enter your password so it can finish installing:

6. When Docker is finished, a little whale icon will show up in the status menu in the top right of your screen, as shown in the following screenshot:

7. If you click on the whale, you can access the application preferences and other options.
8. Select the About Docker button to verify that you have the latest version.
9. Check to make sure that it is installed and working. Open up a terminal window and type the following:

            $ docker container run hello-world

The preceding recipe will download and install a Docker development environment on your Mac.

Now that you have Docker for Mac installed, here are a few more tips for getting started:

• Docker for Mac will automatically check for new updates and let you know when a new version is available for you to install. If you agree to upgrade, it will do all the work, downloading the new version and installing it for you.
• Kubernetes isn't running by default. If you want to turn it on, you will need to click on the Docker whale icon in your Status menu, then select Preferences. Inside of Preferences, there is a Kubernetes tab. Click on the tab, then click the Enable Kubernetes option, and hit the Apply button:

Visit the Get Started with Docker for Mac guide to help you learn about the application and how best to use it. You can find it at https://docs.docker.com/docker-for-mac/.

I am borrowing the following recipe from the next chapter to introduce some concepts. Don't worry if the recipe doesn't explain everything; we'll cover the topics in detail later in this chapter, or in the next few chapters. For now, let's pull an image and run it. We'll also get familiar with Docker architecture and its components in this recipe.

First, gain access to a system that has Docker installed.

To pull an image and run a container, go through the following steps:

1. Pull an image by running the following command:

      $ docker image pull alpine

2. List the existing images by using the following command:

      $ docker image ls

3. Create a container using the pulled image and list the containers as follows:

        $ docker container run -id --name demo alpine ash

Docker has client–server architecture. Its binary consists of the Docker client and server daemon, and can reside on the same host. The client can communicate via sockets or a RESTful API to either a local or remote Docker daemon. The Docker daemon builds, runs, and distributes containers. As shown in the following diagram, the Docker client sends the command to the Docker daemon running on the host machine. The Docker daemon also connects to either a public or a local registry to get images requested by the client:

So in our case, the Docker client sends a request to the daemon running on the local system, which then connects to the public Docker registry and downloads the image. Once it is downloaded, we can run it.

Let's explore some keywords that we encountered earlier in this recipe:

• Images: Docker images are read-only templates, and they give us containers during runtime. They are based on the idea of a base image and layers resting on top of it. For example, we can have a base image of Alpine or Ubuntu, and then we can install packages or make modifications over the base image to create a new layer. The base image and new layer can be treated as a new image. For example, in the following figure, Debian is the base image and then Emacs and Apache are the two layers added on top of it. They are highly portable and can be shared easily:

Layers are transparently laid on top of the base image to create a single coherent filesystem.

• Registries: A registry holds Docker images. It can be public or private, depending on the location from which you can download or upload images. The public Docker registry is called Docker Hub, which we will cover later.
• Index: An index manages user accounts, permissions, searches, tagging, and all that nice stuff that's in the public web interface of the Docker registry.
• Containers: Containers run images that are created by combining the base image and the layers on top of it. They contain everything that is needed to run an application. As shown in the preceding diagram, a temporary layer is also added while starting the container, and this will be discarded if it is not committed after the container is stopped and deleted. If it is committed, then it would create another layer.
• Repository: Different versions of an image can be managed by multiple tags, which are saved with different GUID. A repository is a collection of images tracked by GUIDs.

For more information, check out the documentation on the Docker website at https://docs.docker.com/engine/docker-overview/.

For ease of use, we can allow a nonroot user to administer Docker by adding it to a Docker group. This is not required when using Docker on Mac or Windows.

To prepare to add a nonroot user to administer Docker, go through the following steps:

1. Create the Docker group, if it is not there already:

    $ sudo groupadd docker

2. Create the user to whom you want to give permissions to administer Docker:

    $ sudo useradd dockertest

Run the following command to add the newly created user to administer Docker:

    $ sudo usermod -aG docker dockertest

The preceding command will add a user to the Docker group. The added user will thus be able to perform all Docker operations.

Docker commands are well documented, and can be referred to whenever needed. Lots of documentation is available online as well, but it might differ from the documentation for the Docker version you are running.

First, install Docker on your system.

1. On a Linux-based system, you can use the man command to find help, as follows:

    $ man docker

2. Subcommand-specific help can also be found with either of the following commands:

    $ man docker ps
    $ man docker-ps

The man command uses the man pages installed by the Docker package to provide help.

For more information, see the documentation on the Docker website at https://docs.docker.com/engine/reference/commandline/cli/.