3. Multiprocessing: When and How to Use It

When performance is critical and tasks are CPU-bound, multiprocessing is the way to go. Each process runs in its own Python interpreter and has its own memory space.

Multiprocessing is a technique used in programming to perform multiple operations simultaneously. It is particularly useful when your tasks are CPU-bound, meaning they require significant computer processing power. In multiprocessing, each task runs in its own separate Python interpreter. This separation means that each process has its own dedicated memory space, allowing them to operate independently of each other. This is beneficial for performance because it means tasks can truly run in parallel across multiple CPU cores without being hindered by the limitations of Python's Global Interpreter Lock (GIL).

Think of multiprocessing as having multiple chefs in a kitchen, each working on different dishes simultaneously. If all chefs had to share the same workstation (like a single process in threading), they would get in each other's way and slow down the cooking. However, with multiprocessing, each chef has their own space and tools, allowing them to work efficiently at the same time.

from multiprocessing import Process import os def compute(): print(f"Running on process ID: {os.getpid()}") p1 = Process(target=compute) p2 = Process(target=compute) p1.start() p2.start() p1.join() p2.join()

The multiprocessing module in Python provides a straightforward way to create and manage multiple processes. First, we import the Process class from the multiprocessing package. We then define a function, compute, which performs a task, like printing the current process ID using os.getpid(). To create new processes, we instantiate Process objects, passing the target function. We start these processes using the .start() method, which begins their execution. Finally, we call .join() on each process to ensure that the main program waits for them to complete before continuing. This structure allows efficient execution of multiple tasks in parallel.

Imagine you're running a factory where each machine can produce a different part of a product. When you start a machine, it begins working on its task. You can have many machines (processes) running simultaneously, and when all machines finish, you can assemble all the parts. Using join is like waiting for all machines to complete their jobs before moving forward with assembly.

✅ True parallelism using multiple CPU cores ✅ Bypasses the GIL ❌ Higher overhead than threads ❌ Data must be serialized for communication between processes

Multiprocessing offers significant advantages and some drawbacks. The strengths include:

• True parallelism: Multiprocessing can leverage multiple CPU cores effectively, allowing for faster processing of CPU-bound tasks.
• Bypassing the GIL: Unlike threading, multiprocessing avoids the limitations imposed by the Global Interpreter Lock, allowing processes to run in parallel without contention. However, there are also challenges:
• Overhead: Creating processes typically consumes more resources than threads, leading to higher overhead.
• Data Serialization: For processes to communicate, data must be serialized (converted into a format that can be transmitted), which can introduce complexity and potential performance bottlenecks.

Think of multiprocessing as hiring several highly skilled workers (processes) who can work at full capacity on different tasks. This setup enhances productivity, but it comes with costs like higher wages (overhead) and the need for a manager to keep track of progress and ensure everyone understands their tasks (data serialization).

--