Installing C/C++ packages

Most Python packages are purely Python and blissfully easy to install, just as a simple pip install packagename does the trick. However, there are cases where compilation is involved and installation goes from a simple pip install to searching for hours to see which dependencies are needed to install a certain package.

The specific error message will differ as per the project and environment, but there is a common pattern in these errors, and understanding what you are looking at can help a lot when searching for a solution.

For example, when installing pillow on a standard Ubuntu machine, you'll get a few pages full of errors, warnings, and other messages that end like this:

    x86_64-linux-gnu-gcc: error: build/temp.linux-x86_64-3.4/libImaging/Jpeg2KDecode.o: No such file or directory
    x86_64-linux-gnu-gcc: error: build/temp.linux-x86_64-3.4/libImaging/Jpeg2KEncode.o: No such file or directory
    x86_64-linux-gnu-gcc: error: build/temp.linux-x86_64-3.4/libImaging/BoxBlur.o: No such file or directory
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
Command "python3 -c "import setuptools, tokenize;__file__='/tmp/pip-build-_f0ryusw/pillow/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-kmmobum2-record/install-record.txt --single-version-externally-managed --compile --install-headers include/site/python3.4/pillow" failed with error code 1 in /tmp/pip-build-_f0ryusw/pillow

Upon seeing messages like these, you might be tempted to search for one of the lines such as x86_64-linux-gnu-gcc: error: build/temp.linux-x86_64-3.4/libImaging/Jpeg2KDecode.o: No such file or directory. While this might give you some relevant results, most likely it will not. The trick with installations like these is to scroll up until you see messages about missing headers. Here is an example:

  In file included from libImaging/Imaging.h:14:0,
                   from libImaging/Resample.c:16:
  libImaging/ImPlatform.h:10:20: fatal error: Python.h: No such file or directory
   #include "Python.h"
                      ^
  compilation terminated.

The key message here is that Python.h is missing. These are part of the Python headers and are needed for the compilation of most C/C++ packages within Python. Depending on the operating system, the solutions will vary—unfortunately. So, I recommend that you skip all parts of this paragraph that are not relevant for your case.

Debian and Ubuntu

In Debian and Ubuntu, the package to be installed is python3-dev or python2-dev if you're still using Python 2. The command to execute is as follows:

# sudo apt-get install python3-dev

However, this installs the development headers only. If you want the compiler and other headers bundled with the install, then the build-dep command is also very useful. Here is an example:

# sudo apt-get build-dep python3

Red Hat, CentOS, and Fedora

Red Hat, CentOS, and Fedora are rpm-based distros that use the yum package manager to install the requirements. Most development headers are available through <package-name>-devel and are easily installable as such. To install the Python 3 development headers, use this line:

# sudo apt-get install python3-devel

To make sure you have all the requirements such as development headers and compilers to build packages such as Python, the yum-builddep command is available:

# yum-builddep python3

OS X

The install procedure on OS X consists of three steps before the actual package can be installed.

First, you have to install Xcode. This can be done through the OS X App Store at https://itunes.apple.com/en/app/xcode/id497799835?mt=12.

Then you have to install the Xcode command-line tools:

# xcode-select --install

Finally, you need to install the Homebrew package manager. The steps are available at http://brew.sh/, but the install command is as follows:

# /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Note

Other package managers, such as Macports, are also possible, but Homebrew is currently the OS X package manager with the most active development and community.

Once all of these steps have been completed, you should have a working Homebrew installation. The working of Homebrew can be verified using the brew doctor command. If there are no major errors in the output, then you should be ready to install your first packages through brew. Now we simply need to install Python and we're done:

# brew install python3

Windows

On Windows, manual compilation of C Python packages is generally a non-trivial task to say the least. Most packages have been written with Linux/Unix systems in mind (OS X falls under the Unix category), and Windows is a nice-to-have for developers. The result is that packages are difficult to compile on Windows because there are few people testing them and many of the libraries require manual installation, making it a very tedious task. So, unless you really have to, try and stay away from manually compiling Python packages on Windows. Most packages are available as installable binary downloads with a bit of searching, and there are alternatives such as Anaconda that include binary packages for most important C Python packages.

If you still feel inclined to manually compile C Python packages, then there is another option, and it is generally an easier alternative. The Cygwin project (http://cygwin.com/) attempts to make Linux applications run natively on Windows. This is generally an easier solution than making packages work with Visual Studio.

If you do wish to take the Visual Studio path, I would like to point you towards Chapter 14, Extensions in C/C++, System Calls, and C/C++ Libraries, which covers manual writing of C/C++ extensions and some information on which Visual Studio versions you need for your Python version.