Python’s multiprocessing performance problem
Because Python has limited parallelism when using threads, using worker processes is a common way to take advantage of multiple CPU cores.
The multiprocessing module is built-in to the standard library, so it’s frequently used for this purpose.
But while multiple processes let you take advantage of multiple CPUs, moving data between processes can be very slow.
And that can reduce some of the performance benefits of using worker processes.
Let’s see:
- Why processes can have performance problems that threads don’t.
- A number of ways to work around or deal with this performance overhead.
- A bad solution you don’t want to us.
Threads vs. processes
Multiple threads let you run code in parallel, potentially on multiple CPUs.
On Python, however, the global