Working with Docker

Sometimes more flexibility is needed across projects than simply to differentiate Python package versions. For example, it might be necessary to support an application on an existing version of Python itself, or perhaps MySQL, while simultaneously developing an update that relies upon a newer version of the software. Docker is capable of that level of isolation.

Docker is a system for creating configured, customized virtual machines called containers. It allows duplicating the setup of any production server precisely. In some cases, it is even possible to deploy pre-built containers directly to remote servers as well.

Getting ready

First, you will need to install the Docker Engine, following the instructions to be found at https://www.docker.com/get-started. This usually includes the Compose tool, which makes it simple to manage systems that require multiple containers, ideal for a fully isolated Django project. If needed, installation details for Compose are available at https://docs.docker.com/compose/install/.

How to do it...

With Docker and Compose installed, we will start by creating a myproject_docker directory. Within this, create subdirectories named apps, config, media, project, static, and templates. Then, we will create three configuration files:

• A requirements.txt file defining Python dependencies, under the config directory
• Dockerfile for the Django application container, in the myproject_docker root
• A docker-compose.yml file identifying all of the services making up the application environment, also in the myproject_docker root

The requirements.txt, which lives under the config subdirectory, is much the same as if using a virtual environment (see the Working with a virtual environment recipe), though we will include all dependencies here, not just those that differ from other projects. Because we are likely trying to match our Docker environment to that of the production server, we will generally require very specific versions of each module. In this case, we limit to the latest patch within a minor version range. For example, here, we would prefer mysqlclient 1.3.13 over mysqlclient 1.3.3, but we would not yet upgrade to mysqlclient 1.4.0:

# config/requirements.txt
Pillow~=5.2.0
mysqlclient~=1.3.0
Django~=2.1.0

Dockerfile will define how to build the environment within the container:

# Dockerfile
FROM python:3
RUN apt-get update \
    && apt-get install -y --no-install-recommends \
        mysql-client libmysqlclient-dev
WORKDIR /usr/src/app
ADD config/requirements.txt ./
RUN pip3 install --upgrade pip; \
    pip3 install -r requirements.txt
RUN django-admin startproject myproject .; \
    mv ./myproject ./origproject

We start with the official image for Python 3, install some dependencies for MySQL, set our working directory, add and install Python requirements, and then start a Django project.

Finally, docker-compose.yml puts together the Django application container with other services, such as a MySQL database, so that we can run them together with ease:

# docker-compose.yml
version: '3'
services:
  db:
    image: 'mysql:5.7'
  app:
    build: .
    command: python3 manage.py runserver 0.0.0.0:8000
    volumes:
      - './project:/usr/src/app/myproject'
      - './media:/usr/src/app/media'
      - './static:/usr/src/app/static'
      - './templates:/usr/src/app/templates'
      - './apps/external:/usr/src/app/external'
      - './apps/myapp1:/usr/src/app/myapp1'
      - './apps/myapp2:/usr/src/app/myapp2'
    ports:
      - '8000:8000'
    links:
      - db

As we can see in the volumes section, we will also need to add subdirectories within myproject_docker named project, media, static, and templates, plus each of the apps for the project. These directories will house the code, configuration, and other resources that are exposed within the container.

How it works...

With our basic configuration in place, we can now issue commands to Docker to build and start up our services. If the system we built was using only Dockerfile, this could be done without Compose, using direct docker engine commands. However, in a Compose setup there is a special docker-compose wrapper command that makes it easier to coordinate multiple interconnected containers.

The first step is to build our containers, as defined by the docker-compose.yml file. The first time that you build, any images used as starting points need to be loaded locally, and then each instruction in the Dockerfile is performed sequentially within the resultant machine:

myproject_docker/$ docker-compose build
db uses an image, skipping
Building app
Step 1/6 : FROM python:3
3: Pulling from library/python
f49cf87b52c1: Pull complete
7b491c575b06: Pull complete
b313b08bab3b: Pull complete
51d6678c3f0e: Pull complete
09f35bd58db2: Pull complete
0f9de702e222: Pull complete
73911d37fcde: Pull complete
99a87e214c92: Pull complete
Digest: sha256:98149ed5f37f48ea3fad26ae6c0042dd2b08228d58edc95ef0fce35f1b3d9e9f
Status: Downloaded newer image for python:3
 ---> c1e459c00dc3
Step 2/6 : RUN apt-get update && apt-get install -y --no-install-recommends mysql-client libmysqlclient-dev
 ---> Running in 385946c3002f
Get:1 http://security.debian.org jessie/updates InRelease [63.1 kB]
Ign http://deb.debian.org jessie InRelease
Get:2 http://deb.debian.org jessie-updates InRelease [145 kB]
Get:3 http://deb.debian.org jessie Release.gpg [2434 B]
Get:4 http://deb.debian.org jessie Release [148 kB]
Get:5 http://security.debian.org jessie/updates/main amd64 Packages [607 kB]
Get:6 http://deb.debian.org jessie-updates/main amd64 Packages [23.1 kB]
Get:7 http://deb.debian.org jessie/main amd64 Packages [9064 kB]
Fetched 10.1 MB in 10s (962 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
  libdbd-mysql-perl libdbi-perl libmysqlclient18 libterm-readkey-perl
  mysql-client-5.5 mysql-common
Suggested packages:
  libclone-perl libmldbm-perl libnet-daemon-perl libsql-statement-perl
The following NEW packages will be installed:
  libdbd-mysql-perl libdbi-perl libterm-readkey-perl mysql-client
  mysql-client-5.5
The following packages will be upgraded:
  libmysqlclient-dev libmysqlclient18 mysql-common
3 upgraded, 5 newly installed, 0 to remove and 8 not upgraded.
Need to get 4406 kB of archives.
After this operation, 39.8 MB of additional disk space will be used.
Get:1 http://security.debian.org/ jessie/updates/main libmysqlclient-dev amd64 5.5.59-0+deb8u1 [952 kB]
Get:2 http://deb.debian.org/debian/ jessie/main libdbi-perl amd64 1.631-3+b1 [816 kB]
Get:3 http://security.debian.org/ jessie/updates/main mysql-common all 5.5.59-0+deb8u1 [80.2 kB]
Get:4 http://deb.debian.org/debian/ jessie/main libdbd-mysql-perl amd64 4.028-2+deb8u2 [119 kB]
Get:5 http://security.debian.org/ jessie/updates/main libmysqlclient18 amd64 5.5.59-0+deb8u1 [674 kB]
Get:6 http://deb.debian.org/debian/ jessie/main libterm-readkey-perl amd64 2.32-1+b1 [28.0 kB]
Get:7 http://security.debian.org/ jessie/updates/main mysql-client-5.5 amd64 5.5.59-0+deb8u1 [1659 kB]
Get:8 http://security.debian.org/ jessie/updates/main mysql-client all 5.5.59-0+deb8u1 [78.4 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 4406 kB in 5s (768 kB/s)
(Reading database ... 21636 files and directories currently installed.)
Preparing to unpack .../libmysqlclient-dev_5.5.59-0+deb8u1_amd64.deb ...
Unpacking libmysqlclient-dev (5.5.59-0+deb8u1) over (5.5.58-0+deb8u1) ...
Preparing to unpack .../mysql-common_5.5.59-0+deb8u1_all.deb ...
Unpacking mysql-common (5.5.59-0+deb8u1) over (5.5.58-0+deb8u1) ...
Preparing to unpack .../libmysqlclient18_5.5.59-0+deb8u1_amd64.deb ...
Unpacking libmysqlclient18:amd64 (5.5.59-0+deb8u1) over (5.5.58-0+deb8u1) ...
Selecting previously unselected package libdbi-perl.
Preparing to unpack .../libdbi-perl_1.631-3+b1_amd64.deb ...
Unpacking libdbi-perl (1.631-3+b1) ...
Selecting previously unselected package libdbd-mysql-perl.
Preparing to unpack .../libdbd-mysql-perl_4.028-2+deb8u2_amd64.deb ...
Unpacking libdbd-mysql-perl (4.028-2+deb8u2) ...
Selecting previously unselected package libterm-readkey-perl.
Preparing to unpack .../libterm-readkey-perl_2.32-1+b1_amd64.deb ...
Unpacking libterm-readkey-perl (2.32-1+b1) ...
Selecting previously unselected package mysql-client-5.5.
Preparing to unpack .../mysql-client-5.5_5.5.59-0+deb8u1_amd64.deb ...
Unpacking mysql-client-5.5 (5.5.59-0+deb8u1) ...
Selecting previously unselected package mysql-client.
Preparing to unpack .../mysql-client_5.5.59-0+deb8u1_all.deb ...
Unpacking mysql-client (5.5.59-0+deb8u1) ...
Setting up mysql-common (5.5.59-0+deb8u1) ...
Setting up libmysqlclient18:amd64 (5.5.59-0+deb8u1) ...
Setting up libmysqlclient-dev (5.5.59-0+deb8u1) ...
Setting up libdbi-perl (1.631-3+b1) ...
Setting up libdbd-mysql-perl (4.028-2+deb8u2) ...
Setting up libterm-readkey-perl (2.32-1+b1) ...
Setting up mysql-client-5.5 (5.5.59-0+deb8u1) ...
Setting up mysql-client (5.5.59-0+deb8u1) ...
Processing triggers for libc-bin (2.19-18+deb8u10) ...
Removing intermediate container 385946c3002f
 ---> 6bca605a6e41
Step 3/6 : WORKDIR /usr/src/app
Removing intermediate container 3b23729581ef
 ---> 75bf10f0bee4
Step 4/6 : ADD config/requirements.txt ./
 ---> 31a62236f4b9
Step 5/6 : RUN pip3 install --upgrade pip; pip3 install -r requirements.txt
 ---> Running in 755a1b397b5d
Requirement already up-to-date: pip in /usr/local/lib/python3.6/site-packages
Collecting Pillow~=5.2.0 (from -r requirements.txt (line 2))
  Downloading Pillow-5.2.0-cp36-cp36m-manylinux1_x86_64.whl (5.9MB)
Collecting mysqlclient~=1.3.0 (from -r requirements.txt (line 3))
  Downloading mysqlclient-1.3.0.tar.gz (76kB)
Collecting Django~=2.1.0 (from -r requirements.txt (line 4))
  Downloading Django-2.1.1-py3-none-any.whl (7.1MB)
Collecting pytz (from Django~=2.1.0->-r requirements.txt (line 4))
  Downloading pytz-2017.3-py2.py3-none-any.whl (511kB)
Building wheels for collected packages: mysqlclient
  Running setup.py bdist_wheel for mysqlclient: started
  Running setup.py bdist_wheel for mysqlclient: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/0e/11/a1/e81644c707456461f470c777f13fbd11a1af8eff0ca71aaca0
Successfully built mysqlclient
Installing collected packages: Pillow, mysqlclient, pytz, Django
Successfully installed Django-2.1.1 Pillow-5.2.0 mysqlclient-1.3.0 pytz-2017.3
Removing intermediate container 755a1b397b5d
 ---> 12308a188504
Step 6/6 : RUN django-admin startproject myproject .; mv ./myproject ./origproject
 ---> Running in 746969588bd3
Removing intermediate container 746969588bd3
 ---> 8bc2b0beb674
Successfully built 8bc2b0beb674
Successfully tagged myprojectdocker_app:latest

This will create a local image based on the code in the myproject_docker directory. We can see a list of the built images available, as follows:

myproject_docker/$ docker images
REPOSITORY           TAG     IMAGE ID      CREATED         SIZE
myprojectdocker_app  latest  6a5c66f22a02  39 seconds ago  814MB
python               3       c1e459c00dc3  4 weeks ago     692MB

The state of the machine, after each step, is cached so that future build commands do as little work as possible, based only on the steps after which a change was made. For example, if we build again right away, then everything should come from the cache:

myproject_docker/$ docker-compose build
db uses an image, skipping
Building app
Step 1/6 : FROM python:3
 ---> c1e459c00dc3
Step 2/6 : RUN apt-get update && apt-get install -y --no-install-recommends mysql-client libmysqlclient-dev
 ---> Using cache
 ---> f2007264e96d
Step 3/6 : WORKDIR /usr/src/app
 ---> Using cache
 ---> 9621b97ef4ec
Step 4/6 : ADD config/requirements.txt ./
 ---> Using cache
 ---> 6a87941c7876
Step 5/6 : RUN pip3 install --upgrade pip; pip3 install -r requirements.txt
 ---> Using cache
 ---> 64a268b8cba6
Step 6/6 : RUN django-admin startproject myproject .; mv ./myproject ./origproject
 ---> Using cache
 ---> 8bc2b0beb674
Successfully built 8bc2b0beb674
Successfully tagged myprojectdocker_app:latest

Although we added a project to the container via the Dockerfile, the project volume set up for the app would mask some files when the container is running. To get around this, we moved the project files within the container aside to an origproject directory. Compose allows us to easily run commands against our services, so we can copy those project files so they are accessible in the volume by executing the following command:

myproject_docker/$ docker-compose run app cp \
> origproject/__init__.py \
> origproject/settings.py \
> origproject/urls.py \
> origproject/wsgi.py \
> myproject/

We can see that the previously masked project files are now exposed for us to easily edit outside of the container, too:

myproject_docker/$ ls project
__init__.py   settings.py   urls.py   wsgi.py

Once our services are built and the Django project is created, we can use docker-compose to bring up the environment, passing an optional -d flag to detach the process from our terminal. Detaching runs the containers in exactly the same way, except we can use the terminal to invoke other commands in the meantime. With the containers attached, we are only able to view logs that are exposed by the container (generally what is output to stdout or stderr). The first time we start our Compose environment, any pure image-based services will also need to be pulled down. For example, we might see something like this:

myproject_docker/$ docker-compose up -d
Creating network "myprojectdocker_default" with the default driver
Pulling db (mysql:5.7)...
5.7: Pulling from library/mysql
f49cf87b52c1: Already exists
78032de49d65: Pull complete
837546b20bc4: Pull complete
9b8316af6cc6: Pull complete
1056cf29b9f1: Pull complete
86f3913b029a: Pull complete
f98eea8321ca: Pull complete
3a8e3ebdeaf5: Pull complete
4be06ac1c51e: Pull complete
920c7ffb7747: Pull complete
Digest: sha256:7cdb08f30a54d109ddded59525937592cb6852ff635a546626a8960d9ec34c30
Creating myprojectdocker_db_1 ... done
Creating myprojectdocker_app_1 ... done

At this point, Django is now accessible, just as it would be when run directly on your machine and accessing http://localhost:8000/:

It is often necessary to execute commands within an already up-and-running container, and Docker provides a simple way to do this, as well. As an example, we can connect to the machine at a command-line prompt, similarly to how we might access a remote machine over SSH, as follows:

myproject_docker/$ docker exec -it myproject_docker_app_1 /bin/bash
root@042bf38a407f:/usr/src/app# ls
db.sqlite3  external     manage.py         media   myapp1    myapp2
myproject   origproject  requirements.txt  static  templates
root@042bf38a407f:/usr/src/app# ls myproject
__init__.py  __pycache__  settings.py  urls.py  wsgi.py
root@042bf38a407f:/usr/src/app# exit
myproject_docker/$

The preceding code instructs Docker to execute /bin/bash on the myprojectdocker_app_1 container. The -i flag makes the connection interactive, and -t allocates a TTY shell. Shutting down is just as easy. If the container is running in attached mode, simply issue a Ctrl-C keyboard command to end the process. When using the -d flag to start the container, however, we instead issue a command to shut it down:

myproject_docker/$ docker-compose down
Stopping myprojectdocker_app_1 ... done
Removing myprojectdocker_app_1 ... done
Removing myprojectdocker_db_1 ... done
Removing network myprojectdocker_default

There's more...

Read more from the extensive documentation of Docker at https://docs.docker.com/, and specifically about using Compose with Django at https://docs.docker.com/compose/django/. In the Creating a Docker project file structure recipe, we also go into greater depth around the organization of files and configuration to replicate a production environment.

See also

• The Working with a virtual environment recipe
• The Creating a Docker project file structure recipe