Inference Optimization

Continuous Batching — Maximizing GPU Utilization

Static batching wastes GPU resources waiting for the longest request to complete. Continuous batching dynamically adds and removes requests during generation, keeping the GPU fully utilized.

Dynamic Scheduling — Add new requests as soon as slots free up
Iteration-Level Scheduling — Make scheduling decisions every generation step
Throughput Optimization — 3-5x throughput improvement over static batching

The GPU should never wait for a request, and no request should wait for the GPU.

Continuous Batching for LLMs

Traditional static batching requires all requests in a batch to complete before processing the next batch. This leads to significant waste — if one request finishes early, its GPU resources sit idle until the entire batch completes.

Static vs Continuous Batching

The Problem with Static Batching

Continuous Batching Solution

Throughput Analysis

Implementation Architecture

import asyncio
from dataclasses import dataclass
from typing import List, Optional
import heapq

@dataclass
class Request:
    request_id: str
    input_ids: torch.Tensor
    max_tokens: int
    priority: int = 0
    tokens_generated: int = 0
    done: bool = False

class ContinuousBatchScheduler:
    def __init__(self, model, max_batch_size=32):
        self.model = model
        self.max_batch_size = max_batch_size
        self.active_requests: List[Request] = []
        self.pending_requests: asyncio.Queue = asyncio.Queue()
    
    async def add_request(self, request: Request):
        await self.pending_requests.put(request)
    
    async def generate(self):
        while True:
            # Fill batch from pending requests
            while len(self.active_requests) < self.max_batch_size:
                if self.pending_requests.empty():
                    break
                request = await self.pending_requests.get()
                self.active_requests.append(request)
            
            if not self.active_requests:
                await asyncio.sleep(0.001)
                continue
            
            # Run one generation step
            input_batch = torch.stack([r.input_ids for r in self.active_requests])
            with torch.no_grad():
                outputs = self.model(input_batch)
            
            # Process each request
            completed = []
            for i, request in enumerate(self.active_requests):
                next_token = sample_token(outputs.logits[i])
                request.input_ids = torch.cat([request.input_ids, next_token.unsqueeze(0)])
                request.tokens_generated += 1
                
                if next_token.item() == EOS_TOKEN or request.tokens_generated >= request.max_tokens:
                    request.done = True
                    completed.append(request)
            
            # Remove completed requests
            for req in completed:
                self.active_requests.remove(req)

Scheduling Policies

First-In-First-Out (FIFO)

class FIFOScheduler:
    def select_next(self, pending, active, max_batch):
        if len(active) >= max_batch:
            return None
        if not pending:
            return None
        return pending.pop(0)

Shortest-Job-First (SJF)

class SJFScheduler:
    def select_next(self, pending, active, max_batch):
        if len(active) >= max_batch:
            return None
        if not pending:
            return None
        # Sort by estimated output length
        pending.sort(key=lambda r: r.estimated_output_length)
        return pending.pop(0)

Preemptive Scheduling

class PreemptiveScheduler:
    def __init__(self, preempt_threshold=0.1):
        self.preempt_threshold = preempt_threshold
    
    def maybe_preempt(self, active_requests, new_request):
        if new_request.priority > min(r.priority for r in active_requests):
            # Preempt lowest priority request
            victim = min(active_requests, key=lambda r: r.priority)
            self.move_to_cpu(victim)
            active_requests.remove(victim)
            active_requests.append(new_request)

Memory Management

KV Cache Management

Production Systems

vLLM Architecture

Performance Comparison

System	Throughput (tokens/s)	Latency (p99)	GPU Utilization
HuggingFace (naive)	1,000	500ms	30-40%
vLLM (continuous)	5,000	200ms	80-90%
TensorRT-LLM	8,000	150ms	85-95%
SGLang	6,000	180ms	82-92%

Practice Exercises

Throughput Analysis: Compare the throughput of static vs continuous batching for a workload with 50% short (<10 tokens) and 50% long (>500 tokens) requests.
Scheduler Design: Implement a priority-based scheduler that minimizes tail latency (p99) while maintaining high throughput.
KV Cache Budget: If you have 80GB GPU memory and the model uses 40GB, how many concurrent requests can you serve with continuous batching?
Preemption Policy: Design a preemption policy that minimizes the number of preempted requests while ensuring high-priority requests complete on time.

Key Takeaways

What to Learn Next

-> KV Cache Optimization Reducing memory usage of the key-value cache.

-> Speculative Decoding Generating multiple tokens per step for faster inference.

-> LLM Inference Optimization Broader strategies for making LLM inference faster.

-> Flash Attention and Memory Efficiency IO-aware attention algorithms.

-> Building Production LLM Applications End-to-end production LLM systems.

-> Model Parallelism and Tensor Parallelism Splitting models across GPUs.

Continuous Batching for LLMs

Continuous Batching — Maximizing GPU Utilization

Continuous Batching for LLMs

Static vs Continuous Batching

The Problem with Static Batching

Continuous Batching Solution

Throughput Analysis

Implementation Architecture

Scheduling Policies

First-In-First-Out (FIFO)

Shortest-Job-First (SJF)

Preemptive Scheduling

Memory Management

KV Cache Management

Production Systems

vLLM Architecture

Performance Comparison

Practice Exercises

Key Takeaways

What to Learn Next

Need Expert LLM Help?