Distributing Python code across multiple cores with IPython

Despite CPython's GIL, it is possible to execute several tasks in parallel on multi-core computers using multiple processes instead of multiple threads. Python offers a native multiprocessing module. IPython offers an even simpler interface that brings powerful parallel computing features in an interactive environment. We will describe this tool here.

How to do it…

1. First, we launch four IPython engines in separate processes. We have basically two options to do this:

   • Executing ipcluster start -n 4 in a system shell

   • Using the web interface provided in the IPython notebook's main page by clicking on the Clusters tab and launching four engines

2. Then, we create a client that will act as a proxy to the IPython engines. The client automatically detects the running engines:

   In [2]: from IPython.parallel import Client
           rc = Client()

3. Let's check the number of running engines:

   In [3]: rc.ids
   Out[3]: [0, 1, 2, 3]

4. To run commands in parallel over...