Sharing data between threads and processes
Data sharing is really the most difficult part about multiprocessing, multithreading, and distributed programming in general: which data to pass along, which data to share, and which data to skip. The theory is really simple, however: whenever possible, don’t transfer any data, don’t share any data, and keep everything local. This is essentially the functional programming paradigm, which is why functional programming mixes really well with multiprocessing. In practice, regrettably, this is simply not always possible. The multiprocessing library has several options to share data, but internally they break down to two different options:
- Shared memory: This is by far the fastest solution since it has very little overhead, but it can only be used for immutable types and is restricted to a select few types and custom objects that are created through
multiprocessing.sharedctypes. This is a fantastic solution if you only...
