Python Concurrency — Threading & Multiprocessing

Concurrency lets you do multiple things at once. Python offers threading (for I/O-bound) and multiprocessing (for CPU-bound) tasks.

Learning Objectives

• Use threading for I/O-bound tasks
• Use multiprocessing for CPU-bound tasks
• Understand the GIL and its implications
• Apply ThreadPoolExecutor and ProcessPoolExecutor

Threading Basics

import threading
import time

def download(url):
    print(f"Downloading {url}")
    time.sleep(2)
    return f"Data from {url}"

urls = ["url1", "url2", "url3"]
threads = []
for url in urls:
    t = threading.Thread(target=download, args=(url,))
    threads.append(t)
    t.start()

for t in threads:
    t.join()

ThreadPoolExecutor

from concurrent.futures import ThreadPoolExecutor
import requests

def fetch(url):
    response = requests.get(url, timeout=10)
    return response.status_code

urls = ["https://example.com"] * 10

with ThreadPoolExecutor(max_workers=5) as executor:
    results = list(executor.map(fetch, urls))

Multiprocessing

from multiprocessing import Pool
import math

def compute(n):
    return sum(math.factorial(i) for i in range(n))

with Pool() as pool:
    results = pool.map(compute, [1000, 2000, 3000, 4000])

GIL — Global Interpreter Lock

# GIL means only ONE thread executes Python bytecode at a time
# Threading is good for I/O-bound (network, disk, sleep)
# Multiprocessing is good for CPU-bound (math, data processing)

# CPU-bound with threading (NO speedup due to GIL)
# CPU-bound with multiprocessing (SPEEDUP from multiple cores)

Key Takeaways

1. Use ThreadPoolExecutor for I/O-bound tasks
2. Use ProcessPoolExecutor for CPU-bound tasks
3. GIL limits threading to one core for Python code
4. Multiprocessing bypasses GIL by using separate processes
5. Use asyncio for high-concurrency network operations