Multiprocessing

Introduction

Multiprocessing enables true parallelism by using multiple processes, bypassing GIL limitations.

Basic Multiprocessing

import multiprocessing as mp

def worker(n):
    return n ** 2

if __name__ == "__main__":
    with mp.Pool(4) as pool:
        results = pool.map(worker, range(10))
    print(results)  # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Process Communication

# Shared memory
value = mp.Value("i", 0)
array = mp.Array("i", [0, 0, 0])

# Queue
q = mp.Queue()
q.put("item")
item = q.get()

# Pipe
parent_conn, child_conn = mp.Pipe()

Process Pool Methods

with mp.Pool(4) as pool:
    # Map
    results = pool.map(func, items)
    
    # Async map
    async_result = pool.map_async(func, items)
    results = async_result.get()
    
    # Apply
    result = pool.apply(func, (arg,))
    
    # Starmap
    results = pool.starmap(func, [(1,2), (3,4)])

Practice Problems

1. Parallelize matrix multiplication
2. Use Pool with callback functions
3. Share data between processes
4. Implement parallel file processing
5. Use imap_unordered for streaming results