Multiple threads and processes
The multiprocessing
module was introduced in Python 2.6, and it has been a game changer when it comes to working with multiple processes in Python. Specifically, it has made it rather easy to work around the limitations of the GIL because each process has its own GIL.
The usage of the multiprocessing
module is largely similar to the threading
module, but it has several really useful extra features that make much more sense with multiple processes. Alternatively, you can also use it with concurrent.futures.ProcessPoolExecutor
, which has an interface nearly identical to concurrent.futures.ThreadPoolExecutor
.
These similarities mean that in many cases you can simply swap out the modules and your code will keep running as expected. Don’t be fooled, however; while threads can still use the same memory objects and only have thread safety and deadlocks to worry about, multiple processes also have these issues and introduce several other issues...