Using a message queue to transmit objects

The multiprocessing module uses serialization and transmission of objects, too. We can use queues and pipes to serialize objects that are then transmitted to other processes. There are numerous external projects to provide sophisticated message queue processing. We'll focus on the multiprocessing queue because it's built-in to Python and works nicely.

For high-performance applications, a faster message queue may be necessary. It may also be necessary to use a faster serialization technique than pickling. For this chapter, we'll focus only on the Python design issues. The multiprocessing module relies on pickle to encode objects. See Chapter 9, Serializing and Saving – JSON, YAML, Pickle, CSV, and XML, for more information. We can't provide a restricted unpickler easily; therefore, this module offers us some relatively simple security measures put into place to prevent unpickle problems.

There is one important design consideration when using multiprocessing...