Object Detection with YOLO and Faster R-CNN

Module: Computer Vision | Difficulty: Advanced

Detection as Regression and Classification

Object detection simultaneously solves two problems: localizing objects with bounding boxes and classifying them into categories. Unlike image classification which outputs a single label for the entire image, detection must output a variable-length list of predictions, each containing bounding box coordinates, a class label, and a confidence score. This variable-length output requirement is one of the fundamental challenges distinguishing detection from classification.

Modern detectors follow one of two paradigms: two-stage detectors first generate region proposals (candidate object locations) and then classify each proposal, while one-stage detectors directly predict boxes and classes from dense anchor grids. The trade-off is between accuracy (two-stage) and speed (one-stage), though recent advances like YOLOv8 and DETR have largely closed this gap. The field has evolved from sliding window approaches to learning-based proposal generation, and most recently to set prediction formulations that eliminate hand-designed components entirely.

The loss function for detection combines classification and regression objectives:

Where each parameter means:

• — classification loss (cross-entropy or focal loss)
• — bounding box regression loss (smooth L1 or GIoU)
• — objectness loss (binary cross-entropy)
• — balancing weights (typically 1.0 and 1.0)
• Intuition: Detection requires jointly learning what objects are present and where they are; the three losses balance these competing objectives

Intersection over Union (IoU)

IoU measures the overlap between predicted and ground-truth bounding boxes, serving as the primary metric for detection quality:

Where each parameter means:

• — predicted bounding box
• — ground-truth bounding box
• — area of intersection between boxes
• — area of union between boxes
• Intuition: IoU ranges from 0 (no overlap) to 1 (perfect overlap); a detection is considered correct if IoU exceeds a threshold (typically 0.5)

Anchor Box Regression

Detectors predict offsets from predefined anchor boxes to match ground-truth objects:

Where each parameter means:

• — normalized x and y offsets
• — log-scale width and height adjustments
• — anchor box center and dimensions
• — ground-truth box center and dimensions
• Intuition: Normalizing by anchor dimensions makes the regression targets scale-invariant, so the same network can detect small and large objects

Loss Functions

Focal Loss

Focal loss addresses class imbalance in one-stage detectors by down-weighting easy negatives:

Where each parameter means:

• — model's predicted probability for the ground-truth class
• — balancing factor (typically 0.25 for positive, 0.75 for negative)
• — focusing parameter (typically 2.0)
• Intuition: Easy examples (high ) get near-zero loss, while hard examples dominate the gradient; this prevents thousands of easy negatives from overwhelming the learning signal

Non-Maximum Suppression (NMS)

NMS removes redundant detections by suppressing overlapping boxes:

Where each parameter means:

• — confidence score of detection
• — detection with highest score
• — IoU threshold (typically 0.5)
• Intuition: For each class, keep the highest-scoring detection and suppress all others that overlap significantly, preventing the same object from being detected multiple times

GIoU Loss

Generalized IoU provides better gradients than standard IoU for box regression:

Where each parameter means:

• — smallest enclosing box containing both and
• — standard intersection over union
• Intuition: GIoU adds a penalty for the area between the two boxes that is not covered by either, providing gradients even when boxes don't overlap

Detection Model Comparison

Complete YOLO Inference Pipeline

import torch
import torch.nn as nn
import torchvision.ops as ops


class YOLOv8Head(nn.Module):
    def __init__(self, in_channels, num_classes, num_anchors=9):
        super().__init__()
        self.num_classes = num_classes
        self.num_anchors = num_anchors
        self.cls_head = nn.Conv2d(in_channels, num_anchors * num_classes, 1)
        self.box_head = nn.Conv2d(in_channels, num_anchors * 4, 1)
        self.obj_head = nn.Conv2d(in_channels, num_anchors, 1)

    def forward(self, features):
        batch_size = features.shape[0]
        cls_pred = self.cls_head(features)
        box_pred = self.box_head(features)
        obj_pred = self.obj_head(features)
        cls_pred = cls_pred.permute(0, 2, 3, 1).reshape(batch_size, -1, self.num_classes)
        box_pred = box_pred.permute(0, 2, 3, 1).reshape(batch_size, -1, 4)
        obj_pred = obj_pred.permute(0, 2, 3, 1).reshape(batch_size, -1, 1)
        return box_pred, cls_pred, obj_pred


def compute_iou_matrix(boxes1, boxes2):
    x1 = torch.max(boxes1[:, None, 0], boxes2[None, :, 0])
    y1 = torch.max(boxes1[:, None, 1], boxes2[None, :, 1])
    x2 = torch.min(boxes1[:, None, 2], boxes2[None, :, 2])
    y2 = torch.min(boxes1[:, None, 3], boxes2[None, :, 3])
    intersection = (x2 - x1).clamp(0) * (y2 - y1).clamp(0)
    area1 = (boxes1[:, 2] - boxes1[:, 0]) * (boxes1[:, 3] - boxes1[:, 1])
    area2 = (boxes2[:, 2] - boxes2[:, 0]) * (boxes2[:, 3] - boxes2[:, 1])
    union = area1[:, None] + area2[None, :] - intersection
    return intersection / (union + 1e-6)


def non_maximum_suppression(boxes, scores, iou_threshold=0.5, score_threshold=0.25):
    mask = scores > score_threshold
    boxes = boxes[mask]
    scores = scores[mask]
    if len(boxes) == 0:
        return torch.zeros(0, dtype=torch.long)
    order = scores.argsort(descending=True)
    keep = []
    while order.numel() > 0:
        i = order[0].item()
        keep.append(i)
        if order.numel() == 1:
            break
        ious = compute_iou_matrix(boxes[i:i+1], boxes[order[1:]])[0]
        mask = ious < iou_threshold
        order = order[1:][mask]
    return torch.tensor(keep, dtype=torch.long)


def decode_predictions(boxes, strides, image_size=640):
    grid_y, grid_x = torch.meshgrid(
        torch.arange(boxes.shape[1]), torch.arange(boxes.shape[1]), indexing='ij'
    )
    grid = torch.stack([grid_x, grid_y], dim=-1).float()
    boxes[..., 0:2] = (boxes[..., 0:2].sigmoid() + grid) * strides
    boxes[..., 2:4] = boxes[..., 2:4].exp() * strides
    return boxes

Common Challenges

1. Scale Variation: Objects in images range from tiny (10 pixels) to huge (1000+ pixels), requiring multi-scale feature pyramids
2. Class Imbalance: Training datasets have severe class imbalance, with thousands of easy negatives overwhelming few positive examples
3. Crowded Scenes: Overlapping objects create ambiguous bounding boxes, requiring sophisticated NMS or learned suppression
4. Real-Time Constraints: Autonomous driving and robotics require detection at 30+ FPS, limiting model complexity
5. Domain Gap: Models trained on one domain (daytime) often fail on another (nighttime), requiring domain adaptation

Real-World Case Study: Autonomous Driving Detection

Waymo's perception system (2022) processes 200,000 LiDAR points and 6 camera images per frame at 10 Hz, detecting 200+ object types including vehicles, pedestrians, and cyclists. Their multi-modal detector achieves 72.3% mAP@50 on the Waymo Open Dataset (1,150 scenes), with 99.1% recall for vehicles within 50 meters. The system uses a PointPillars LiDAR backbone fused with a ResNet-101 image backbone through a feature pyramid network. Latency per frame is 65ms on a custom TPU, with the NMS stage consuming 12ms. The key engineering challenge was handling 100K+ proposals per frame efficiently through a hierarchical NMS that first suppresses LiDAR-only proposals, then camera-only, and finally cross-modal.

Key Takeaways

• Object detection combines localization (regression) with classification in a single model
• Anchor boxes provide scale and aspect ratio priors that improve recall
• Focal loss addresses the extreme class imbalance inherent in dense detection
• Feature Pyramid Networks enable multi-scale detection by combining semantic and spatial features
• NMS is essential for removing redundant predictions but adds inference latency
• Modern anchor-free detectors achieve competitive accuracy with simpler architectures