Handling project dependencies with pip

The most convenient tool to install and manage Python packages is pip. Rather than installing the packages one by one, it is possible to define a list of packages that you want to install as the contents of a text file. We can pass the text file into the pip tool, which will then handle installation of all packages in the list automatically. An added benefit to this approach is that the package list can be stored in version control. If you have gone through the Working with Docker recipe, then you have already seen this.

Generally speaking, it is ideal and often sufficient to have a single requirements file that directly matches your production environment. When changing versions or adding and removing dependencies, this can be done on a development machine and then managed through version control. It can then be as simple as switching branches to go from one set of dependencies (and associated code changes) to another.

In some cases, environments differ enough that you will need to have at least two different instances of your project: the development environment, where you create new features, and the public website environment that is usually called the production environment, in a hosted server. There might be development environments for other developers, or special tools that are needed during development but are unnecessary in production. Also, you may have a testing and staging environment in order to test the project locally and in a public website-like situation.

For good maintainability, you should be able to install the required Python modules for development, testing, staging, and production environments. Some of the modules will be shared and some of them will be specific to a subset of the environments. In this recipe, we will see how to organize the project dependencies for multiple environments and manage them with pip.

Getting ready

Before using this recipe, you need to have a Django project ready, either with pip installed and a virtual environment activated, or via Docker. For more information on how to do this, read the Working with a virtual environment recipe, or the Working with Docker recipe, respectively.

How to do it...

Execute the following steps one by one to prepare pip requirements for your virtual environment Django project:

1. Let's go to your Django project that you have under version control and create a requirements directory with these text files, if you haven't already done so:
   • base.txt for shared modules
   • dev.txt for the development environment
   • test.txt for the testing environment
   • staging.txt for the staging environment
   • prod.txt for production

2. Edit base.txt and add the Python modules that are shared in all environments, line by line. For example, we might migrate our original requirements.txt as base.txt, which would give us this in our virtual environment project:

# base.txt
Django~=2.1.0
djangorestframework
-e git://github.com/omab/python-social-auth.git@6b1e301c79#egg=python-social-auth

3. If the requirements of a specific environment are the same as in base.txt, add the line including base.txt in the requirements file of that environment, as in the following example:

# prod.txt
-r base.txt

4. If there are specific requirements for an environment, add them after the base.txt inclusion, as shown in the following:

# dev.txt
-r base.txt
django-debug-toolbar
selenium

5. You can run the following command in a virtual environment in order to install all of the required dependencies for the development environment (or analogous command for other environments), as follows:

(myproject_env)$ pip3 install -r requirements/dev.txt

With a Docker setup, we follow steps 1-4 in almost precisely the same manner, except the requirements directory would live underneath the config directory. From there, a few additional steps are needed to install the correct requirements by environment:

1. The Dockerfile file will need to be updated to select the appropriate requirements file based on a build argument, which here defaults to prod:

# Dockerfile
FROM python:3
RUN apt-get update \
    && apt-get install -y --no-install-recommends \
        less mysql-client libmysqlclient-dev
WORKDIR /usr/src/app
ARG BUILD_ENV=prod
ADD config/requirements ./requirements
RUN pip3 install --upgrade pip; \
    pip3 install -r requirements/$BUILD_ENV.txt
RUN django-admin startproject myproject .; \
    mv ./myproject ./origproject

2. The docker-compose.yml file needs to pass through this argument using the current environment variable, as in the following:

# docker-compose.yml
version: '3'
services:
  db:
    image: "mysql:5.7"
  app:
    build:
      context: .
      args:
        BUILD_ENV: $BUILD_ENV
    command: "python3 manage.py runserver 0.0.0.0:8000"
    volumes:
      - "./project:/usr/src/app/myproject"
      - "./media:/usr/src/app/media"
      - "./static:/usr/src/app/static"
      - "./templates:/usr/src/app/templates"
      - "./apps/external:/usr/src/app/external"
      - "./apps/myapp1:/usr/src/app/myapp1"
      - "./apps/myapp2:/usr/src/app/myapp2"
    ports:
      - "8000:8000"
    depends_on:
      - db

3. Scripts under bin for each environment are then updated to set the appropriate value for the BUILD_ENV variable. For example, we would update the dev script as follows:

#!/usr/bin/env bash
# bin/dev
# ...

BUILD_ENV="dev" \adds
#...
  docker-compose $*

4. We simply use the environment-specific script when building the container, and the argument passes through automatically, causing the correct requirements file to be added to the container:

myproject_docker/$ MYSQL_USER=myproject_user \
> MYSQL_PASSWORD=pass1234 \
> ./bin/dev build

How it works...

The preceding pip3 install command, whether it is executed explicitly in a virtual environment or during the build process for a Docker container, downloads and installs all of your project dependencies from requirements/base.txt and requirements/dev.txt. As you can see, you can specify a version of the module that you need for the Django framework and even directly install from a specific commit at the Git repository, as done for social-app-django in our example.

In practice, installing from a specific commit would rarely be useful, for instance, only when having third-party dependencies in your project, with specific functionality, that are not supported in any other versions.

When you have many dependencies in your project, it is good practice to stick to a narrow range of release versions for Python module release versions. Then you can have greater confidence that the project integrity will not be broken due to updates in your dependencies, which might contain conflicts or backward incompatibility. This is particularly important when deploying your project or handing it off to a new developer.

If you have already manually installed the project requirements with pip one by one, you can generate the requirements/base.txt file using the following command within your virtual environment:

(myproject_env)$ pip3 freeze > requirements/base.txt

The same can be executed within the Docker app container, as in the following:

myproject_docker/$ docker exec -it myproject_docker_app_1 \
> /bin/bash
root:/usr/src/app# pip3 freeze > requirements/base.txt

There's more...

If you want to keep things simple and are sure that, for all environments, you will be using the same dependencies, you can use just one file for your requirements named requirements.txt, generated by definition, as in the following:

(myproject_env)$ pip3 freeze > requirements.txt

To install the modules in a new virtual environment, simply call the following command:

(myproject_env)$ pip3 install -r requirements.txt

If you need to install a Python library from other version control system, or at a local path, you can learn more about pip from the official documentation at http://pip.readthedocs.org/en/latest/reference/pip_install.html.

See also

• The Working with a virtual environment recipe
• The Working with Docker recipe
• The Including external dependencies in your project recipe
• The Configuring settings for development, testing, staging, and production environments recipe