Deploying a first application

We have created a few Docker Swarms on various platforms. Once created, a Swarm behaves the same way on any platform. The way we deploy and update applications on a Swarm is not platform-dependent. It has been one of Docker’s main goals to avoid vendor lock-in when using a Swarm. Swarm-ready applications can be effortlessly migrated from, say, a Swarm running on-premises to a cloud-based Swarm. It is even technically possible to run part of a Swarm on-premises and another part in the cloud. It works, yet we have, of course, to consider possible side effects due to the higher latency between nodes in geographically distant areas.

Now that we have a highly available Docker Swarm up and running, it is time to run some workloads on it. I’m using the swarm just created on AWS. We’ll start by first creating a single service. For this, we need to SSH into one of the manager nodes. I selected the swarm node on the manager1 instance:

$ ssh -i "aws-docker-demo.pem" <public-dns-name-of-manager1>

We start the deployment of our first application to the swarm by creating a service.

Creating a service

A service can be created either as part of a stack or directly using the Docker CLI. Let’s first look at a sample stack file that defines a single service:

1. Use the Vi editor to create a new file called stack.yml and add this content:

   version: "3.7"
   services:
     whoami:
       image: training/whoami:latest
       networks:
       - test-net
       ports:
       - 81:8000
       deploy:
         replicas: 6
         update_config:
           parallelism: 2
           delay: 10s
         labels:
            app: sample-app
            environment: prod-south
   networks:
     test-net:
       driver: overlay

2. Exit the Vi editor by first pressing the Esc key, then typing :wq, and then pressing Enter. This will save the code snippet and exit vi.

Note

If you are not familiar with Vi, you can also use nano instead.

In the preceding example, we see what the desired state of a service called whoami is:

• It is based on the training/whoami:latest image
• Containers of the service are attached to the test-net network
• The container port 8000 is published to port 81
• It is running with six replicas (or tasks)
• During a rolling update, the individual tasks are updated in batches of two, with a delay of 10 seconds between each successful batch
• The service (and its tasks and containers) is assigned the two labels, app and environment, with the values sample-app and prod-south, respectively

There are many more settings that we could define for a service, but the preceding ones are some of the more important ones. Most settings have meaningful default values. If, for example, we do not specify the number of replicas, then Docker defaults it to 1. The name and image of a service are, of course, mandatory. Note that the name of the service must be unique in the Swarm.

3. To create the preceding service, we use the docker stack deploy command. Assuming that the file in which the preceding content is stored is called stack.yaml, we have the following:

   $ docker stack deploy -c stack.yaml sample-stack

   Here, we have created a stack called sample-stack that consists of one service, whoami.

4. We can list all stacks on our Swarm:

   $ docker stack ls

   Upon doing so, we should get this:

   NAME                       SERVICES
   sample-stack            1

5. We can list the services defined in our Swarm, as follows:

   $ docker service ls

   We get the following output:

Figure 14.14 – List of all services running in the Swarm

In the output, we can see that currently, we have only one service running, which was to be expected. The service has an ID. The format of the ID, contrary to what you have used so far for containers, networks, or volumes, is alphanumeric (in the latter cases, it was always SHA-256). We can also see that the name of the service is a combination of the service name we defined in the stack file and the name of the stack, which is used as a prefix. This makes sense since we want to be able to deploy multiple stacks (with different names) using the same stack file into our Swarm. To make sure that service names are unique, Docker decided to combine the service name and stack name.

In the third column, we see the mode, which is replicated. The number of replicas is shown as 6/6. This tells us that six out of the six requested replicas are running. This corresponds to the desired state. In the output, we also see the image that the service uses and the port mappings of the service.

Inspecting the service and its tasks

In the preceding output, we cannot see the details of the six replicas that have been created.

To get some deeper insight into that, we can use the docker service ps <service-id> command. If we execute this command for our service, we will get the following output:

Figure 14.15 – Details of the whoami service

In the preceding output, we can see the list of six tasks that corresponds to the requested six replicas of our whoami service. In the NODE column, we can also see the node to which each task has been deployed. The name of each task is a combination of the service name plus an increasing index. Also note that, similar to the service itself, each task gets an alphanumeric ID assigned.

In my case, apparently tasks 3 and 6, with the names sample-stack_whoami.3 and sample-stack_whoami.6, have been deployed to ip-172-31-32-21, which is the leader of our Swarm. Hence, I should find a container running on this node. Let’s see what we get if we list all containers running on ip-172-31-32-21:

Figure 14.16 – List of containers on node ip-172-31-32-21

As expected, we find a container running from the training/whoami:latest image with a name that is a combination of its parent task name and ID. We can try to visualize the whole hierarchy of objects that we generated when deploying our sample stack:

Figure 14.17 – Object hierarchy of a Docker Swarm stack

A stack can consist of one-to-many services. Each service has a collection of tasks. Each task has a one-to-one association with a container. Stacks and services are created and stored on the Swarm manager nodes. Tasks are then scheduled to Swarm worker nodes, where the worker node creates the corresponding container. We can also get some more information about our service by inspecting it. Execute the following command:

$ docker service inspect sample-stack_whoami

This provides a wealth of information about all of the relevant settings of the service. This includes those we have explicitly defined in our stack.yaml file, but also those that we didn’t specify and that, therefore, got their default values assigned. We’re not going to list the whole output here, as it is too long, but I encourage you to inspect it on your own machine. We will discuss part of the information in more detail in the The swarm routing mesh section in Chapter 15.

Testing the load balancing

To see that the swarm load balances incoming requests to our sample whoami application, we can use the curl tool. Execute the following command a few times and observe how the answer changes:

$ for i in {1..7}; do curl localhost:81; done

This results in an output like this:

I'm ae8a50b5b058
I'm 1b6b507d900c
I'm 83864fb80809
I'm 161176f937cf
I'm adf340def231
I'm e0911d17425c
I'm ae8a50b5b058

Note that after the sixth item, the sequence is repeating. This is due to the fact that the Docker swarm is load balancing calls using a round-robin algorithm.

Logs of a service

In an earlier chapter, we worked with the logs produced by a container. Here, we’re concentrating on a service. Remember that, ultimately, a service with many replicas has many containers running. Hence, we would expect that, if we asked the service for its logs, Docker would return an aggregate of all logs of those containers belonging to the service. And indeed, we’ll see this when we use the docker service logs command:

$ docker service logs sample-stack_whoami

This is what we get:

Figure 14.18 – Logs of the whoami service

There is not much information in the logs at this point, but it is enough to discuss what we get. The first part of each line in the log always contains the name of the container combined with the node name from which the log entry originates. Then, separated by the vertical bar (|), we get the actual log entry. So, if we were to, say, ask for the logs of the first container in the list directly, we would only get a single entry, and the value we would see in this case would be Listening on :8000.

The aggregated logs that we get with the docker service logs command are not sorted in any particular way. So, if the correlation of events is happening in different containers, you should add information to your log output that makes this correlation possible.

Typically, this is a timestamp for each log entry. But this has to be done at the source; for example, the application that produces a log entry needs to also make sure a timestamp is added.

We can also query the logs of an individual task of the service by providing the task ID instead of the service ID or name. So, say we queried the logs from task 6 with the following:

$ docker service logs w90b8

This gives us the following output:

sample-stack_whoami.6.w90b8xmkdw53@ip-172-31-32-21    | Listening on :8000

In the next section, we are investigating how the swarm reconciles the desired state.

Reconciling the desired state

We have learned that a Swarm service is a description or manifest of the desired state that we want an application or application service to run in. Now, let’s see how Docker Swarm reconciles this desired state if we do something that causes the actual state of the service to be different from the desired state. The easiest way to do this is to forcibly kill one of the tasks or containers of the service.

Let’s do this with the container that has been scheduled on node-1:

$ docker container rm -f sample-stack_whoami.3. nqxqs...

If we do that and then run docker service ps right afterward, we will see the following output:

Figure 14.19 – Docker Swarm reconciling the desired state after one task failed

We see that task 2 failed with exit code 137 and that the Swarm immediately reconciled the desired state by rescheduling the failed task on a node with free resources. In this case, the scheduler selected the same node as the failed tasks, but this is not always the case. So, without us intervening, the Swarm completely fixed the problem, and since the service is running in multiple replicas, at no time was the service down.

Let’s try another failure scenario. This time, we’re going to shut down an entire node and are going to see how the Swarm reacts. Let’s take node ip-172-31-47-124 for this, as it has two tasks (tasks 1 and 4) running on it. For this, we can head over to the AWS console and in the EC2 dashboard, stop the instance called ip-172-31-47-124.

Note that I had to go into the details of each worker instance to find out which one has the hostname ip-172-31-47-124; in my case, it was worker2.

Back on the master node, we can now again run docker service ps to see what happened:

Figure 14.20 – Swarm reschedules all tasks of a failed node

In the preceding screenshot, we can see that immediately, task 1 was rescheduled on node ip-172-31-32-189, while task 4 was rescheduled on node ip-172-31-32-21. Even this more radical failure is handled gracefully by Docker Swarm.

It is important to note though that if node ip-172-31-47-124 ever comes back online in the Swarm, the tasks that had previously been running on it will not automatically be transferred back to it.

But the node is now ready for a new workload.

Deleting a service or a stack

If we want to remove a particular service from the Swarm, we can use the docker service rm command. If, on the other hand, we want to remove a stack from the Swarm, we analogously use the docker stack rm command. This command removes all services that are part of the stack definition. In the case of the whoami service, it was created by using a stack file and hence we’re going to use the latter command:

$ docker stack rm sample-stack

This gives us this output:

Removing service sample-stack_whoami
Removing network sample-stack_test-net

The preceding command will make sure that all tasks of each service of the stack are terminated, and the corresponding containers are stopped by first sending SIGTERM, and then, if not successful, SIGKILL after 10 seconds of timeout.

It is important to note that the stopped containers are not removed from the Docker host.

Hence, it is advised to purge containers from time to time on worker nodes to reclaim unused resources. Use docker container purge -f for this purpose.

Question: Why does it make sense to leave stopped or crashed containers on the worker node and not automatically remove them?

Deploying a multi-service stack

In Chapter 11, Managing Containers with Docker Compose, we used an application consisting of two services that were declaratively described in a Docker Compose file. We can use this Compose file as a template to create a stack file that allows us to deploy the same application into a Swarm:

1. Create a new file called pets-stack.yml, and add this content to it:

   version: "3.7"
   services:
     web:
       image: fundamentalsofdocker/ch11-web:2.0
       networks:
       - pets-net
       ports:
       - 3000:3000
       deploy:
         replicas: 3
     db:
       image: fundamentalsofdocker/ch11-db:2.0
       networks:
       - pets-net
       volumes:
       - pets-data:/var/lib/postgresql/data
   volumes:
     pets-data:
   networks:
     pets-net:
       driver: overlay

   We request that the web service has three replicas, and both services are attached to the overlay network, pets-net.

2. We can deploy this application using the docker stack deploy command:

   $ docker stack deploy -c pets-stack.yml pets

   This results in this output:

   Creating network pets_pets-net
   Creating service pets_db
   Creating service pets_web

   Docker creates the pets_pets-net overlay network and then the two services, pets_web and pets_db.

3. We can then list all of the tasks in the pets stack:

Figure 14.21 – List of all of the tasks in the pets stack

4. Finally, let’s test the application using curl to retrieve an HTML page with a pet. And, indeed, the application works as expected, as the expected page is returned:

Figure 14.22 – Testing the pets application using curl

The container ID is in the output, where it says Delivered to you by container d01e2f1f87df. If you run the curl command multiple times, the ID should cycle between three different values. These are the IDs of the three containers (or replicas) that we have requested for the web service.

5. Once we’re done, we can remove the stack with docker stack rm pets.

Once we’re done with the swarm in AWS, we can remove it.

Removing the swarm in AWS

To clean up the Swarm in the AWS cloud and avoid incurring unnecessary costs, we can use the following command:

$ for NODE in `seq 1 5`; do
    docker-machine rm -f aws-node-${NODE}
done

Next, let’s summarize what we have learned in this chapter.