The code for this book is hosted on GitHub at https://github.com/PacktPublishing/A-Developer-s-Essential-Guide-to-Docker-Compose. In case of an update to the code, it will be updated on GitHub.

Docker Compose is a tool for defining and running multi-container Docker applications. The configuration is achieved using YAML files, and through the Docker Compose CLI utility, we can provision and perform operations on the containers managed by Docker Compose.

Here is a list of features that Compose offers:

• Complex multi-container applications on a single host
• The isolation of Docker workloads
• Bootstrapping and the distribution of complex applications
• Multiple environments
• The ability to preserve data on application change
• The ability to update application versions
• Environment composition
• Reusable configurations
• The simulation of complex production environments
• The deployment of production applications

In this book, we will dive into the preceding features extensively, evaluate how we can benefit from them, and incorporate them into our development process. In the next section, we will install Docker and Compose on our workstation using the operating system of our choice.

Both Docker Compose and the Compose CLI are built using Go. Compose can be run on the three major operating systems: Linux, Windows, and macOS. Since Compose is about managing multi-container Docker applications, the prerequisite is to have Docker installed.

Docker Desktop

On Mac and Windows, Docker Desktop is an installation option. Docker Desktop handles the complexity of setting up Docker on your local machine. It will create a Linux Virtual Machine (VM) on your host and facilitate container interactions with the OS such as access to the filesystem and networking. This one-click installation comes with the necessary tools such as the Docker CLI. One of the tools that is included is also Docker Compose. Therefore, installing Docker Desktop makes it sufficient to interact with Docker Engine using Compose on our workstation.

Installing Docker

To install the correct Docker distribution for the workstation of our choice, we will navigate to the corresponding section of the official Docker page:

• Docker Desktop for Mac: https://docs.docker.com/desktop/mac/install/
• Docker Desktop for Windows: https://docs.docker.com/desktop/windows/install/
• Docker Engine for Linux: https://docs.docker.com/engine/install/

On macOS

Apple provides workstations with two different types of processors: an Intel processor and an Apple processor. Docker has an installation option for both. Once the download is complete, by clicking on the installer, you can drag and drop the Docker application, as shown in the following screenshot:

Figure 1.1 – Installing Docker on Mac

Once Docker has been installed, we can run a hello world command check:

$ docker run --rm hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
93288797bd35: Pull complete
Digest: sha256:97a379f4f88575512824f3b352bc03cd75e239179eea 0fecc38e597b2209f49a
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
This message shows that your installation appears to be working correctly.
..

Additionally, we have to check whether Compose has been installed:

$ docker compose version
Docker Compose version v2.2.3

Now, let’s look at how to install Docker Desktop on Windows.

On Windows

Similar to Mac, Docker Desktop is installed seamlessly onto your OS ready to be used.

Once you download the EXE installation file and click on it, Docker will be installed along with its utilities. Once this is done, some extra configurations will need to be applied to enable virtualization for Windows.

Whether the backend that’s being used is WSL 2 backend or Hyper-V, you have to set up your machine BIOS to enable virtualization, as shown in the following screenshot:

Figure 1.2 – Enabling virtualization on Windows via BIOS

Once you have logged in to Windows, you will need to enable the corresponding virtualization features.

For WSL 2, you should enable the Virtual Machine Platform feature and the Windows Subsystem for Linux feature:

Figure 1.3 – Enabling virtualization for WSL 2

For Hyper-V you should enable Hyper-V:

Figure 1.4 – Enabling virtualization for Hyper-V

Before you get started, make sure that your user account is added to the docker-users group. Once done, log out from Windows and log in again. You can start Docker, and then you can execute your first Docker command on PowerShell, as follows:

PS C:\Users\my-user> docker run -d -p 80:80 docker/getting-started
Unable to find image 'docker/getting-started:latest' locally
latest: Pulling from docker/getting-started
59bf1c3509f3: Pull complete                                     8d6ba530f648: Pull complete                                      5288d7ad7a7f: Pull complete                                  39e51c61c033: Pull complete                                    ee6f71c6f4a8: Pull complete                                    f2303c6c8865: Pull complete                                    0645fddcff40: Pull complete                                               d05ee95f5d2f: Pull complete                                     Digest: sha256:aa945bdff163395d3293834697fa91fd4c725f47093ec499 f27bc032dc1bdd16
Status: Downloaded newer image for docker/getting-started:latest
852371fcb34fddfe900bddc669af3a7aaab8743f8555fbb9952904bd2516a e7a
PS C:\Users\my-user>

Let’s also check whether Docker Compose has been installed:

PS C:\Users\my-user> docker compose version
Docker Compose version v2.2.3

Next, we will look at how to install Docker Desktop on Linux.

On Linux

At the time of writing, a Docker Desktop installation for Linux is not available, but it’s on the roadmap, and it’s just a matter of time before it’ll be available for Linux. However, Docker Engine is sufficient in order to use Docker Compose.

The most common method of installation is to add the Docker repositories to your Linux workstation and then install Docker Community Edition using the corresponding package manager of the distribution used.

If you have an older version of Docker, you should remove and install the new docker-ce and docker-ce-cli versions. We will assume that this is the first Docker installation on the workstation we are currently using.

Since Red Hat-based Linux distributions are very popular for both workstations and production usage, we will install Docker on Fedora, which is a Red Hat-based distribution.

First, install the dnf-plugins-core package since it contains tools that can assist us with the management of the dnf repositories:

$ sudo dnf -y install dnf-plugins-core

Then, add the docker-ce repo to access the binaries provided by Docker:

$ sudo dnf config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo

Now that the repo has been set up, we can add the packages:

$ sudo dnf install docker-ce docker-ce-cli containerd.io -y

Docker is a daemon that will run as a service to our machine. Therefore, the systemctl commands apply to Docker running as a device:

$ sudo systemctl start docker

Let’s run a hello-world example:

$ sudo docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
…

As you can see, we had to use sudo in almost every command. This can be fixed by having a group called docker, in which users will have the permission to interact with Docker Engine. On the installation of Docker Engine, this group will be created:

$ sudo groupadd docker
$ sudo usermod -aG docker $USER
$ docker run hello-world

Once installed, everything is set up to install Compose on Linux.

We will proceed with the installation link at https://docs.docker.com/compose/install/#install-compose-on-linux-systems:

$ sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
$ docker-compose —version
docker-compose version 1.29.2, build 5becea4c

Here, we can observe that this is an older version of Compose compared to the ones that we saw earlier. There isn’t a standard way to install Compose V2 on Linux, for instance, by installing Docker Desktop on Mac and Windows. However, since it’s feasible to install Compose V2 on Linux, we will proceed in doing so, allowing us to focus on Compose V2.

We will follow the guidelines from the official documentation at https://docs.docker.com/compose/cli-command/#install-on-linux:

$ mkdir -p ~/.docker/cli-plugins/
$ curl -SL https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-linux-x86_64 -o ~/.docker/cli-plugins/docker-compose
$ chmod +x ~/.docker/cli-plugins/docker-compose
$ docker compose version
Docker Compose version v2.2.3

docker compose versus docker-compose

One observation to be made by navigating to the installation instructions for Linux is that a Python version of docker compose has been installed.

Also, this same version can be found on a Windows installation if you try to use the docker-compose command on Windows:

PS C:\Users\my-user> docker-compose-v1.exe version
docker-compose version 1.29.2, build 5becea4c
docker-py version: 5.0.0
CPython version: 3.9.0
OpenSSL version: OpenSSL 1.1.1g  21 Apr 2020
PS C:\Users\my-user> 

The initial Docker Compose was built in Python; therefore, the installation instructions referenced the installation of pip packages.

Note that for new installations of Docker Desktop, the docker-compose command is an alias to docker compose.

The initial version of Compose’s docker-compose is still supported and maintained. In the case of Compose applications built and run using docker-compose, there are supporting tools available such as Compose Switch (https://docs.docker.com/compose/cli-command/#compose-switch) for a smooth migration.

By installing Compose Switch, the old docker-compose command will be replaced by the compose-switch command.

Compose Switch will interpret the command that should have been passed to docker-compose. Then, it will translate it into a command that can be executed by Compose V2. Then, it will invoke Compose V2 using that command.

In this book, we shall focus on Compose V2 since it's part of docker-cli. This is the default on Docker Desktop, has the latest features, and comes with extra commands.

By now, you should have Docker and Docker Compose installed on your workstation and know how to execute some basic commands. You should also understand the previous Compose version and how you can transition to the latest version. Next, we’re going to take a deeper dive into how Compose works and how it interacts with Docker Engine.

Since we have Docker and Docker Compose installed onto our system, let’s take some time and understand what Compose is and how it works behind the scenes.

On GitHub, we can find a project (https://github.com/docker/compose) where the Docker Compose source code is being hosted. By navigating to the source code, we can see and understand more about Compose, as follows:

• Compose integrates with the Docker CLI as a plugin.
• Compose interacts with Docker Engine through the API.
• Compose provides a CLI and its actions translate into Docker Engine API calls.
• Compose will read the Compose YAML file and generate resources accordingly.
• Compose provides a layer for converting docker-compose commands into CLI-compliant ones.
• Compose will interact with Docker objects and distinguish between them using labels.

The Docker CLI provides an API to create and load plugins. Once a plugin has been created and loaded on its invocation, the CLI command will be passed to it:

func pluginMain() {
    plugin.Run(func(dockerCli command.Cli) *cobra.Command {
      …
      }
}
func main() {
    if commands.RunningAsStandalone() {
            os.Args = append([]string{"docker"}, compatibility.Convert(os.Args[1:])...)
    }
    pluginMain()
}

The CLI is based on Cobra (https://github.com/spf13/cobra), which is a popular Go library for CLI applications.

Compose, being a plugin of the Docker CLI, will use a Docker Engine API client provided by the Docker CLI:

lazyInit.WithService(compose.NewComposeService(dockerCli.Client(), dockerCli.ConfigFile()))

Each command passed to the Docker Compose plugin will lead to an interaction with the Docker Engine API on our host. For example, the internals of the ls command:

func (s *composeService) List(ctx context.Context, opts api.ListOptions) ([]api.Stack, error) {
    list, err := s.apiClient.ContainerList(ctx, moby.ContainerListOptions{
        Filters: filters.NewArgs(hasProjectLabelFilter()),
        All:     opts.All,
    })
    if err != nil {
        return nil, err
    }
    return containersToStacks(list)
}

We now have a good understanding of how Compose works and interacts with Docker Engine. You can also refer to the source code for more information. Next, we’re going to run our first Docker Compose application.

Imagine a scenario of wanting to run a static page on a server. For this task, an NGINX server is a good choice. We have a simple HTML file on the static-site/index.html path:

<!DOCTYPE html>
<html>
    <head>
        <title>Hello World</title>
    </head>
    <body>
       <p>Hi! This application should run on docker-compose</p>
    </body>
</html>

By using Docker, we will run an NGINX server using the official image found at https://www.docker.com/blog/how-to-use-the-official-nginx-docker-image/:

$ docker run --rm -p 8080:80 --name nginx-compose nginx

Let’s break this down a little bit:

• Docker Engine will run a Docker NGINX image.
• The default port on the image is 80, so we shall map it locally to 8080 to avoid using a privileged port.
• The name we assign will be constant in order to make interactions with the container easier.
• By using the —rm argument, we ensure that once we are done with our task and stop the container, the container will be deleted.

Our container is up and running. In a different Terminal session, we should access the default NGINX page:

$ curl 127.0.0.1:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Since we have successfully run NGINX, we need to adapt our command in order to use the customized HTML page. A simple and fast way to do this is to mount the file at the path of a container. Let’s exit the previous command using Ctrl + C and then refine the previous command:

docker run --rm -p 8080:80 --name nginx-compose -v $(pwd)/static-site:/usr/share/nginx/html nginx

As expected, the page changes to the one we have specified:

$ curl localhost:8080/index.html
<!DOCTYPE html>
<html>
    <head>
        <title>Hello World</title>
    </head>
    <body>
       <p>Hi! This application should run on docker-compose</p>
    </body>
</html>
$

Now we have everything needed to migrate this application to Compose. We will create a Compose file for the default NGINX installation:

services:
  nginx:
    image: nginx
    ports:
      - 8080:80

Let’s break down what we just did:

• The name of the service will be NGINX.
• The image is the same NGINX image.
• The ports are the same ports used previously.

The content shall be saved to a file named docker-compose.yaml.

Next, we will execute the Compose command on the Terminal:

$ docker compose up
[+] Running 2/0
  Network chapter1_default    Created                          0.0s
  Container chapter1-nginx-1  Created                          0.0s
Attaching to chapter1-nginx-1
chapter1-nginx-1  | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
chapter1-nginx-1  | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
…
$

As expected, the result from the HTTP request is the same as the one that we experienced by just running the Docker container.

The naming of the file is important. We did execute the Compose command to spin up the Compose file, but we did not specify the file to be used. As it happens with docker build and Dockerfile, by running docker compose in a directory, Compose will search for a file named docker-compose.yaml. If the file exists, it’ll be picked up as the default Compose file. Be aware that we are not limited to just one filename; we can use a different filename for our Compose applications. In the following chapters, there are cases where we can use a different name for the Compose files and run the application using the –f option.

Next, we shall mount the custom HTML page through the Compose configuration:

services:
  nginx:
    image: nginx
    ports:
      - 8080:80
    volumes:
      - ./static-site:/usr/share/nginx/html

As simple as our previous Docker command seemed to be, behind the scenes, it created a Docker volume pointing to a path of our filesystem and then it was attached to the container. The same applies to Compose. We specify a volume that points to our filesystem. Then, based on our location, it is mounted to a directory of the container:

$  curl localhost:8080/index.html
<!DOCTYPE html>
<html>
    <head>
        <title>Hello World</title>
    </head>
    <body>
        <p>Hi! This application should run on docker-compose</p>
    </body>
</html>

As expected, the result is the same one with the result of the Docker example.

To review this section, we ran an NGINX instance using Docker CLI and made the transition to Compose by adding the corresponding YAML sections for the Docker command parameters that were used. Now, we are ready to move on to the next stage of this chapter’s journey, where we’ll build and run a Docker image on Docker Compose.

By using Compose, we have achieved running the default NGINX image and changing the default HTML page that was displayed. Since we have started utilizing Compose, we will proceed with using and testing custom Docker images.

For our use case, we want to develop an NGINX image that prints logs in JSON format since it’s feasible for tools such as CloudWatch (https://aws.amazon.com/cloudwatch/), StackDriver (https://cloud.google.com/products/operations), and ELK Stack (https://www.elastic.co/elastic-stack/) to persist data in JSON format and offer enhanced querying capabilities by having field conditions based on JSON elements.

The problem will require us to identify how NGINX defines the current logging format. Since we have a container already running through Compose, we will shell into the container and check the configuration:

$  docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED       STATUS       PORTS                  NAMES
dc0ca7ebe0cb   nginx     "/docker-entrypoint.…"   7 hours ago   Up 7 hours   0.0.0.0:8080->80/tcp   chapter1-nginx-1
$ docker exec -it chapter1-nginx-1 cat /etc/nginx/nginx.conf
user  nginx;
worker_processes  auto;
error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;
events {
    worker_connections  1024;
}
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

By finding our running container using docker ps and issuing cat, through the container shell, we retrieved the current log_format from the instance by checking the /etc/nginx/nginx.conf file. We will change this format to JSON and build a custom Docker image preloaded with that format.

We will copy the file locally to apply the change:

$  docker cp chapter1-nginx-1:/etc/nginx/nginx.conf nginx.conf

By editing nginx.conf instead of log_format, we set the json format:

log_format  main escape=json '{"remote_addr":"$remote_addr","remote_user":"$remote_user","time":"[$time_local]","request":"$request",'
                     '"status":"$status","body_bytes_sent":"$body_bytes_sent","http_referer":"$http_referer",'
                      '"http_user_agent":"$http_user_agent","http_x_forwarded_for":"$http_x_forwarded_for"}';             

Our file will look like this:

user  nginx;
worker_processes  auto;
error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;
events {
    worker_connections  1024;
}
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    log_format  main  escape=json '{"remote_addr":"$remote_addr","remote_user":"$remote_user","time":"[$time_local]","request":"$request",'
                      '"status":"$status","body_bytes_sent":"$body_bytes_sent","http_referer":"$http_referer",'
                      '"http_user_agent":"$http_user_agent","http_x_forwarded_for":"$http_x_forwarded_for"}';
    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    #tcp_nopush     on;
    keepalive_timeout  65;
    #gzip  on;
    include /etc/nginx/conf.d/*.conf;
}

Now that we have the config file needed, we will create the base NGINX image that will use this configuration. The Dockerfile will be the following:

FROM nginx
COPY nginx.conf /etc/nginx/nginx.conf

Let’s build the image:

$ docker build -t custom-nginx:0.1 .

Let’s go ahead and use it with the recently created docker-compose.yaml file:

services:
  nginx:
    image: custom-nginx:0.1
    ports:
      - 8080:80
    volumes:
      - ./static-site:/usr/share/nginx/html
$ docker compose up
…
chapter1-nginx-1  | 2022/02/10 08:09:27 [notice] 1#1: start worker process 33
chapter1-nginx-1  | {"remote_addr":"172.19.0.1","remote_user":"","time":"[10/Feb/2022:08:09:33 +0000]","request":"GET / HTTP/1.1","status":"200","body_bytes_sent":"177","http_referer":"","http_user_agent":"curl/7.77.0","http_x_forwarded_for":""}
…

By now, Compose runs successfully on your application that also uses the custom Docker image. So far, Compose was sufficient to use a custom image and also include some modification at runtime such as mounting a file as well as doing port mapping. The results were the same as the ones we would expect if we run the application using Docker commands.

In this chapter, we were introduced to Docker Compose and some of its most notable features. We installed Compose on different operating systems and identified the differences between installations. Then, we identified the different Compose versions, Docker-Compose V1 and Docker Compose V2, along with the version to be used throughout this book. By checking on the Compose source code, we went a step further regarding how Compose works and interacts with the Docker CLI. Then, we ran a Docker application using the docker-cli command and created the equivalent of it on Compose. The next step was to customize the image we used in our first example and deploy it using Compose.

In the next chapter, we shall create an application that will run and interact with a Redis database using Compose.