Why does multiprocessing avoid the GIL problem, and what overhead does it introduce instead?
Separate processes have separate GILs; the cost is startup, pickling, and IPC
Each Python process has its own interpreter, its own heap, and therefore its own GIL. Two processes can execute Python bytecode on two cores simultaneously with no shared lock, which is how multiprocessing achieves true CPU parallelism. The costs are real: process startup is much more expensive than thread startup, especially on Windows and macOS where spawn is the only start method and the module must be re-imported. Data passed between processes is pickled, so large objects and functions with unpicklable state are problematic. Results come back through pipes or shared memory, and communication has latency. Shared state must use multiprocessing primitives (Value, Array, Manager, Queue) or be avoided entirely in favor of message passing.
- 1
Start methods: fork on Linux (default until 3.13 moved toward forkserver), spawn on Windows and macOS. spawn re-imports the module.
- 2
Pickling: arguments and return values are serialized. Lambdas, open files, and connections cannot be sent.
- 3
IPC: Queue and Pipe are convenient but slower than in-process calls. Shared memory (multiprocessing.shared_memory) avoids copy for large buffers.
- 4
Pool and ProcessPoolExecutor amortize process startup across many tasks.
- 5
Trade-off: true parallelism versus serialization and process overhead. Small tasks can be slower than sequential.
- 6
Common mistake: assuming a global variable is shared across processes. Each process has its own copy.
- 7
Common mistake: forgetting the if name == 'main' guard with spawn, causing infinite process spawning.
- 8
Version note: fork became non-default on Linux in 3.14 for safety with threads. 3.8 added multiprocessing.shared_memory for zero-copy buffers.
Scenario Questions
0-2 years experience
- 1You update a global variable in a child process and the parent does not see it. Why?
- 2Which is faster to start, a thread or a process?
2-5 years experience
- 1You send a lambda to a ProcessPoolExecutor and get a PicklingError. What do you do?
- 2Your script spawns processes infinitely on macOS. What guard is missing?
5-8 years experience
- 1You need to process 1 GB arrays across processes without copying. How do you structure it?
- 2Your multiprocessing job is slower than the sequential version. What overheads do you check?
8+ years experience
- 1Design a multiprocessing pipeline with bounded queues, back-pressure, crash recovery, and shared memory for large payloads.
- 2Explain how fork interacts with threads, locks, and at-fork handlers, and how to make a forking service safe.
Follow-up Questions
- Why did Python move away from fork as the default start method?
- When would you use shared_memory instead of a Queue?