Using a multiprocessing pool for concurrent processing
One elegant way to make use of the multiprocessing
module is to create a processing Pool
object and assign work to the various processes in that pool. We will use the OS to interleave execution among the various processes. If each of the processes has a mixture of I/O and computation, we should be able to ensure that our processor is very busy. When processes are waiting for the I/O to complete, other processes can do their computations. When an I/O finishes, a process will be ready to run and can compete with others for processing time.
The recipe for mapping work to a separate process looks like this:
import multiprocessing with multiprocessing.Pool(4) as workers: workers.map(analysis, glob.glob(pattern))
We've created a Pool
object with four separate processes and assigned this Pool
object to the workers
variable. We've then mapped a function, analysis
, to an iterable queue of work to be done, using the pool of processes...