Computer Vision Pipeline
Object Detection + OCR + FastAPI | GPU-Accelerated Serving
What You'll Learn
- 1Stage detection, cropping, and OCR so each model receives the input format it was built for
- 2Explain non-maximum suppression with IoU and choose a threshold that fits the scene
- 3Export a PyTorch detector to ONNX or TensorRT and measure speedup against accuracy loss
- 4Bound batch endpoints and queue heavy jobs so latency targets hold under load
Project Overview
Problem Statement
Organizations need to extract structured information from images at scale - detecting objects, reading text, and classifying scenes. This pipeline combines state-of-the-art object detection (YOLOv8) with OCR (Tesseract/EasyOCR) into a single production-ready API.
Objectives
- Build an end-to-end CV pipeline with YOLOv8 + OCR
- Implement batch processing for high throughput
- Deploy with GPU acceleration and model optimization
- Create a scalable API with async processing
- Add monitoring and quality metrics tracking
| Component | Technology |
|---|---|
| Object Detection | YOLOv8 (Ultralytics) |
| OCR Engine | Tesseract + EasyOCR |
| API Framework | FastAPI + gunicorn |
| Image Processing | OpenCV + Pillow |
| Model Optimization | ONNX Runtime + TensorRT |
| Queue System | Celery + Redis |
| Storage | MinIO / S3 |
| Monitoring | Prometheus + Grafana |
Design Rationale
Detection and OCR are staged rather than fused because the two models want different inputs. YOLOv8 sees the whole letterboxed frame at 640 x 640 so objects keep their context, while Tesseract wants tight, high-contrast crops at native resolution; running OCR over the full image reads background texture as characters and spends time on regions with no text at all. The pipeline detects first, crops with a 10 pixel pad, then reads each region.
Overlapping detections are resolved by intersection over union:
Non-maximum suppression keeps a box only when its IoU against every higher-scoring box stays below 0.5. Lowering the threshold removes more duplicates but starts merging objects that genuinely overlap, such as a person holding a cup, so 0.5 is a reasonable default for general scenes and 0.3 suits dense aerial imagery.
Latency is a sum over stages, and batching changes the shape of that sum:
A single 640 x 640 frame costs about 8 ms of GPU inference on a T4; a batch of 8 costs about 22 ms, so per-image inference falls from 8 ms to 2.75 ms and throughput rises from 125 FPS to 364 FPS. The gain stops once the GPU is saturated, which is why queue depth rather than replica count is the correct autoscaling signal.
Exporting to ONNX and then TensorRT removes Python overhead from the hot path:
Typical measurements on the same YOLOv8m checkpoint are 2.4x for ONNX Runtime with the CUDA execution provider and 3.5x for TensorRT, with mAP@50 dropping less than 0.5 points. The decision metric is accuracy per millisecond, not raw speed.
Detection quality itself is summarized by mAP@50, the mean over classes of average precision evaluated at IoU 0.5:
A jump from 0.70 to 0.75 mAP usually means better labels or a larger model rather than cleverer post-processing, which is why the metrics table tracks mAP and OCR accuracy next to latency instead of only timing the endpoint.
Architecture Diagram
Step-by-Step Implementation
Step 1: Environment Setup
mkdir cv-pipeline && cd cv-pipeline
python -m venv venv && source venv/bin/activate
pip install ultralytics opencv-python-headless pillow
pip install fastapi uvicorn python-multipart
pip install pytesseract easyocr onnxruntime-gpu
pip install celery[redis] minio python-dotenv
pip install prometheus-client mlflow
# Install Tesseract
sudo apt-get install tesseract-ocr tesseract-ocr-eng
Step 2: Object Detection Module
Implement YOLOv8-based object detection with custom post-processing.
# src/detection/yolo_detector.py
from ultralytics import YOLO
import numpy as np
from typing import List, Dict, Tuple
from dataclasses import dataclass
import cv2
import logging
logger = logging.getLogger(__name__)
@dataclass
class Detection:
class_id: int
class_name: str
confidence: float
bbox: Tuple[int, int, int, int] # x1, y1, x2, y2
area: int
class YOLODetector:
def __init__(self, model_path: str = "yolov8m.pt", confidence: float = 0.5):
self.model = YOLO(model_path)
self.confidence = confidence
logger.info(f"Loaded YOLO model from {model_path}")
def detect(
self, image: np.ndarray,
classes: List[int] = None,
) -> List[Detection]:
results = self.model(
image,
conf=self.confidence,
classes=classes,
verbose=False,
)
detections = []
for r in results:
for box in r.boxes:
x1, y1, x2, y2 = map(int, box.xyxy[0].tolist())
detections.append(Detection(
class_id=int(box.cls[0]),
class_name=r.names[int(box.cls[0])],
confidence=float(box.conf[0]),
bbox=(x1, y1, x2, y2),
area=(x2-x1) * (y2-y1),
))
return detections
def detect_batch(self, images: List[np.ndarray]) -> List[List[Detection]]:
results = self.model(images, conf=self.confidence, verbose=False)
batch_detections = []
for r in results:
dets = []
for box in r.boxes:
x1, y1, x2, y2 = map(int, box.xyxy[0].tolist())
dets.append(Detection(
class_id=int(box.cls[0]),
class_name=r.names[int(box.cls[0])],
confidence=float(box.conf[0]),
bbox=(x1, y1, x2, y2),
area=(x2-x1) * (y2-y1),
))
batch_detections.append(dets)
return batch_detections
def crop_detections(
self, image: np.ndarray, detections: List[Detection],
padding: int = 10
) -> List[Tuple[Detection, np.ndarray]]:
h, w = image.shape[:2]
crops = []
for det in detections:
x1, y1, x2, y2 = det.bbox
x1 = max(0, x1 - padding)
y1 = max(0, y1 - padding)
x2 = min(w, x2 + padding)
y2 = min(h, y2 + padding)
crop = image[y1:y2, x1:x2]
crops.append((det, crop))
return crops
Step 3: OCR Module
Implement OCR extraction from detected regions using both Tesseract and EasyOCR.
# src/ocr/ocr_engine.py
import pytesseract
import easyocr
import numpy as np
from typing import List, Dict, Optional
from dataclasses import dataclass
import cv2
@dataclass
class OCRResult:
text: str
confidence: float
bbox: Optional[tuple] = None
engine: str = "tesseract"
class OCREngine:
def __init__(self, engines: List[str] = None):
self.engines = engines or ["tesseract", "easyocr"]
if "easyocr" in self.engines:
self.easyocr_reader = easyocr.Reader(["en"], gpu=True)
def extract_tesseract(self, image: np.ndarray) -> OCRResult:
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) if len(image.shape) == 3 else image
data = pytesseract.image_to_data(gray, output_type=pytesseract.Output.DICT)
texts = []
confidences = []
for i, text in enumerate(data["text"]):
if text.strip() and int(data["conf"][i]) > 30:
texts.append(text)
confidences.append(int(data["conf"][i]))
combined = " ".join(texts)
avg_conf = sum(confidences) / len(confidences) if confidences else 0
return OCRResult(text=combined, confidence=avg_conf, engine="tesseract")
def extract_easyocr(self, image: np.ndarray) -> OCRResult:
results = self.easyocr_reader.readtext(image)
texts = []
confidences = []
for (bbox, text, conf) in results:
if conf > 0.3:
texts.append(text)
confidences.append(conf)
combined = " ".join(texts)
avg_conf = sum(confidences) / len(confidences) if confidences else 0
return OCRResult(text=combined, confidence=avg_conf, engine="easyocr")
def extract(self, image: np.ndarray) -> OCRResult:
results = []
if "tesseract" in self.engines:
results.append(self.extract_tesseract(image))
if "easyocr" in self.engines:
results.append(self.extract_easyocr(image))
# Return best result by confidence
return max(results, key=lambda r: r.confidence) if results else OCRResult(text="", confidence=0)
def extract_from_crops(self, crops: List[tuple]) -> List[Dict]:
results = []
for detection, crop_image in crops:
ocr_result = self.extract(crop_image)
results.append({
"detection": detection,
"ocr": ocr_result,
})
return results
Step 4: Pipeline Orchestrator
Combine detection and OCR into a unified pipeline with quality scoring.
# src/pipeline/processor.py
import numpy as np
from typing import Dict, List, Any
from dataclasses import dataclass, asdict
import time
import cv2
from src.detection.yolo_detector import YOLODetector
from src.ocr.ocr_engine import OCREngine
@dataclass
class PipelineResult:
detections: List[Dict]
ocr_results: List[Dict]
processing_time_ms: float
image_size: tuple
quality_score: float
class CVPipeline:
def __init__(self, detector: YOLODetector, ocr: OCREngine):
self.detector = detector
self.ocr = ocr
def process_image(self, image: np.ndarray) -> PipelineResult:
start = time.time()
h, w = image.shape[:2]
# Step 1: Detect objects
detections = self.detector.detect(image)
# Step 2: Crop detected regions
crops = self.detector.crop_detections(image, detections)
# Step 3: OCR on cropped regions
ocr_results = self.ocr.extract_from_crops(crops)
# Step 4: Compute quality score
quality = self._compute_quality(detections, ocr_results)
elapsed = (time.time() - start) * 1000
return PipelineResult(
detections=[asdict(d) for d in detections],
ocr_results=[{"text": r["ocr"].text, "confidence": r["ocr"].confidence,
"engine": r["ocr"].engine, "class": r["detection"].class_name}
for r in ocr_results],
processing_time_ms=elapsed,
image_size=(w, h),
quality_score=quality,
)
def _compute_quality(self, detections, ocr_results) -> float:
if not detections:
return 0.0
det_scores = [d.confidence for d in detections]
ocr_scores = [r["ocr"].confidence for r in ocr_results] if ocr_results else [0]
return (np.mean(det_scores) + np.mean(ocr_scores)) / 2
Step 5: FastAPI Service
# src/api/main.py
from fastapi import FastAPI, UploadFile, File, HTTPException
from fastapi.responses import JSONResponse
from pydantic import BaseModel
from typing import List, Optional
from contextlib import asynccontextmanager
import numpy as np
from PIL import Image
import io
import time
from src.pipeline.processor import CVPipeline
from src.detection.yolo_detector import YOLODetector
from src.ocr.ocr_engine import OCREngine
@asynccontextmanager
async def lifespan(app: FastAPI):
detector = YOLODetector("yolov8m.pt", confidence=0.5)
ocr = OCREngine(["tesseract", "easyocr"])
app.state.pipeline = CVPipeline(detector, ocr)
yield
app = FastAPI(title="CV Pipeline API", version="1.0.0", lifespan=lifespan)
@app.post("/api/v1/process")
async def process_image(file: UploadFile = File(...)):
contents = await file.read()
image = Image.open(io.BytesIO(contents))
image_np = np.array(image)
result = app.state.pipeline.process_image(image_np)
return {
"detections": result.detections,
"ocr_results": result.ocr_results,
"processing_time_ms": result.processing_time_ms,
"image_size": result.image_size,
"quality_score": result.quality_score,
}
@app.post("/api/v1/process/batch")
async def process_batch(files: List[UploadFile] = File(...)):
results = []
for file in files[:10]: # Max 10 images per batch
contents = await file.read()
image = Image.open(io.BytesIO(contents))
result = app.state.pipeline.process_image(np.array(image))
results.append({
"filename": file.filename,
"detections": len(result.detections),
"processing_time_ms": result.processing_time_ms,
})
return {"results": results}
Step 6: Model Optimization with ONNX
# src/optimization/export_onnx.py
from ultralytics import YOLO
import onnxruntime as ort
def export_yolo_to_onnx(model_path: str, output_path: str):
model = YOLO(model_path)
model.export(format="onnx", dynamic=True, simplify=True)
print(f"Exported ONNX model to {output_path}")
class ONNXDetector:
def __init__(self, onnx_path: str):
self.session = ort.InferenceSession(
onnx_path,
providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)
def predict(self, input_tensor):
input_name = self.session.get_inputs()[0].name
outputs = self.session.run(None, {input_name: input_tensor})
return outputs
âšī¸
Use ONNX Runtime for 2-4x inference speedup over PyTorch. For even faster inference, export to TensorRT for NVIDIA GPUs.
đĄ
Implement a preprocessing queue with Celery for batch processing. This prevents API timeouts for large images and enables priority-based processing.
Performance Metrics
| Metric | Target | Description |
|---|---|---|
| Detection Latency | < 30ms | Per image (GPU) |
| OCR Latency | < 50ms | Per detected region |
| End-to-End Latency | < 200ms | Full pipeline |
| mAP@50 | > 0.75 | Object detection accuracy |
| OCR Accuracy | > 95% | On clear text |
| Throughput | > 50 FPS | Batch processing |
What Breaks in Production
The first failure is resolution drift. Users upload 4,000 pixel phone photos, naive preprocessing keeps them at full size, and the 30 ms latency target becomes several hundred milliseconds. Letterbox to a fixed 640 long side with aspect ratio preserved, and reject uploads above a pixel budget before decode.
The second is OCR confidence collapse on rotated or low-contrast text. Tesseract often returns plausible-looking garbage with confidence in the 30 to 50 range instead of failing outright, so blindly averaging confidence hides the problem. Gate on it instead:
Crops whose mean confidence falls below 60 are dropped rather than returned, and the rejection rate itself becomes a monitoring signal for camera or scanner quality regressions.
Third, model churn. Constructing YOLODetector inside the request handler adds roughly 300 ms and heavy allocation per call, which is the difference between 50 FPS and 5. Load both models once in the lifespan hook and share them across requests.
Finally, unbounded batch endpoints. Ten 4,000 pixel images in one synchronous request can hold a worker for seconds while the load balancer keeps sending traffic. Cap the synchronous batch at 10 images, push anything larger onto Celery with a priority queue, and answer 202 with a job id so clients can poll instead of timing out.
Interview Talking Points
- YOLO Architecture: Single-shot detection enables real-time inference. YOLOv8 uses anchor-free detection for better generalization.
- OCR Pipeline: Combining object detection with OCR creates a powerful document understanding system.
- Batch Processing: Batching GPU operations provides significant throughput improvements over single-image processing.
- Model Optimization: ONNX Runtime and TensorRT provide 2-5x speedup with minimal accuracy loss.
- Quality Scoring: Confidence-based quality metrics help filter low-quality results for downstream consumers.
- Scaling: GPU instances with auto-scaling based on queue depth for cost-efficient processing.
â ī¸
OCR accuracy degrades significantly with poor image quality. Implement image preprocessing (contrast enhancement, noise reduction) as a first step.
âšī¸
This pipeline handles general object detection and OCR. For specialized use cases (medical imaging, satellite imagery), fine-tune the detection model on domain-specific data.
Common Mistakes
Running OCR over the entire image
Fix: Detect first, crop with a 10 pixel pad, then run OCR per region so background texture is never read as text.
Averaging OCR confidence without a threshold
Fix: Reject crops whose mean confidence is below 60 and track the rejection rate as a quality signal.
Loading models inside the request handler
Fix: Load detector and OCR engine once in the lifespan hook and reuse them across all requests.
Best Practices
- Letterbox uploads to a fixed 640 input so latency is predictable regardless of source resolution
- Track mAP@50 and OCR accuracy alongside latency on every deploy
- Return 202 with a job id for batches above the synchronous pixel budget
Key Takeaways
- IoU is intersection over union, and NMS at 0.5 removes duplicates without merging objects that truly overlap
- Batching 8 frames cuts per-image inference from 8 ms to 2.75 ms, lifting throughput from 125 FPS to 364 FPS
- ONNX Runtime gives about 2.4x and TensorRT about 3.5x over PyTorch with under 0.5 points of mAP loss
- Queue depth, not replica count, is the correct autoscaling signal for GPU workers
Frequently Asked Questions
Which YOLOv8 size should you deploy?
YOLOv8s hits about 45 FPS on a T4 with mAP@50 near 0.76 and fits most real-time budgets. Move to YOLOv8m only when an evaluation shows the smaller model misses classes you care about, because the larger variant costs roughly 60 percent more latency for about 1 point of mAP.
Tesseract or EasyOCR?
Tesseract is faster on clean, high-contrast scans and integrates with a simple system package, while EasyOCR handles stylized, noisy, and scene text better at a higher per-crop cost. Running both and taking the higher-confidence result, as this pipeline does, is reasonable when accuracy matters more than a few milliseconds.
How do you evaluate OCR quality in production?
Use word error rate against a small labeled set of 200 crops, plus the confidence rejection rate as a cheap continuous signal. A sudden jump in rejected crops usually means an upstream change in camera, scanner settings, or image compression rather than a model problem.
Interview Questions
Q1: How does non-maximum suppression work and when does it break?
Sort detections by confidence, keep the top box, and suppress every box with IoU above 0.5 against it, repeating until none remain. It breaks with nested objects like a hand inside a cup, where legitimate boxes overlap heavily and get suppressed.
Q2: Walk me through the latency budget of one image.
Decode and resize about 5 ms, YOLOv8 inference 8 ms on a T4, cropping negligible, OCR about 40 ms per region with three regions at 12 ms each. End-to-end lands near 90 ms, so OCR dominates and batching regions is the lever.
Q3: When do you choose ONNX Runtime over TensorRT?
ONNX Runtime needs no per-GPU engine build and supports dynamic shapes, which suits varying image sizes and fast model iteration. TensorRT pays off in steady-state high volume where its 3.5x speedup over PyTorch justifies a rebuild for every shape and GPU.