Autonomous Driving Perception

Autonomous Driving Perception Overview

Autonomous driving perception fuses data from multiple sensors to understand the 3D environment around the vehicle. The perception stack detects and tracks objects (cars, pedestrians, cyclists), estimates their 3D positions and velocities, and provides a comprehensive world model for planning and control. Multi-sensor fusion combines cameras for semantic understanding with LiDAR for precise depth and geometry.

Theory: Multi-Sensor Fusion

Autonomous driving requires fusing camera, LiDAR, and radar data to compensate for each sensor's limitations. Cameras provide rich semantic information but lack depth; LiDAR gives precise 3D measurements but is sparse and expensive; radar works in all weather but has low resolution. Sensor fusion combines these modalities into a unified representation.

Bird's Eye View (BEV) transformation projects multi-sensor features into a top-down view enabling consistent fusion. Camera features are lifted to 3D using depth estimation, while LiDAR points are projected onto the image plane. The BEV representation preserves spatial relationships and enables efficient convolution-based processing.

The Kalman filter provides optimal state estimation for object tracking. The prediction step propagates the object state forward using a constant velocity model, while the update step corrects the prediction using new measurements. The filter handles uncertainty and provides velocity estimates from position-only measurements.

Mathematical Foundations

The extended Kalman filter for tracking:

Where each parameter means:

• is the predicted state at time
• is the state transition matrix (constant velocity model)
• is the previous updated state

The Kalman gain and update:

Where each parameter means:

• is the Kalman gain matrix
• is the predicted covariance
• is the observation matrix mapping state to measurement
• is the measurement noise covariance

The BEV transformation for camera features:

Where each parameter means:

• is the BEV feature at grid position (u,v)
• is the camera feature at pixel (u,v)
• is the predicted depth distribution for that pixel
• The summation aggregates features along the depth dimension

Architecture Design

Implementation

import torch
import torch.nn as nn
import numpy as np


class KalmanTracker:
    def __init__(self, state_dim=10, meas_dim=5):
        self.state = np.zeros(state_dim)
        self.P = np.eye(state_dim) * 10
        self.F = np.eye(state_dim)
        self.F[0, 4] = 1; self.F[1, 5] = 1; self.F[2, 6] = 1
        self.F[3, 7] = 1
        self.H = np.zeros((meas_dim, state_dim))
        self.H[0, 0] = 1; self.H[1, 1] = 1; self.H[2, 2] = 1
        self.H[3, 3] = 1; self.H[4, 8] = 1
        self.Q = np.eye(state_dim) * 0.1
        self.R = np.eye(meas_dim) * 1.0
        self.age = 0; self.hits = 0; self.id = -1

    def predict(self):
        self.state = self.F @ self.state
        self.P = self.F @ self.P @ self.F.T + self.Q
        self.age += 1

    def update(self, measurement):
        z = measurement
        y = z - self.H @ self.state
        S = self.H @ self.P @ self.H.T + self.R
        K = self.P @ self.H.T @ np.linalg.inv(S)
        self.state = self.state + K @ y
        self.P = (np.eye(len(self.state)) - K @ self.H) @ self.P
        self.hits += 1


class CenterPointDetector(nn.Module):
    def __init__(self, num_classes=3, bev_size=200):
        super(CenterPointDetector, self).__init__()
        self.lidar_backbone = nn.Sequential(
            nn.Conv2d(10, 64, 3, padding=1), nn.BatchNorm2d(64), nn.ReLU(),
            nn.Conv2d(64, 128, 3, stride=2, padding=1), nn.BatchNorm2d(128), nn.ReLU(),
            nn.Conv2d(128, 256, 3, stride=2, padding=1), nn.BatchNorm2d(256), nn.ReLU()
        )
        self.heatmap_head = nn.Sequential(nn.Conv2d(256, 128, 3, padding=1), nn.ReLU(),
                                         nn.Conv2d(128, num_classes, 1))
        self.offset_head = nn.Sequential(nn.Conv2d(256, 128, 3, padding=1), nn.ReLU(),
                                         nn.Conv2d(128, 2, 1))
        self.bbox_head = nn.Sequential(nn.Conv2d(256, 128, 3, padding=1), nn.ReLU(),
                                       nn.Conv2d(128, 10, 1))

    def forward(self, bev_features):
        features = self.lidar_backbone(bev_features)
        heatmap = torch.sigmoid(self.heatmap_head(features))
        offset = self.offset_head(features)
        bbox = self.bbox_head(features)
        return heatmap, offset, bbox


def nms_3d(boxes, scores, iou_threshold=0.2):
    order = scores.argsort()[::-1]
    keep = []
    while len(order) > 0:
        i = order[0]
        keep.append(i)
        if len(order) == 1:
            break
        remaining = order[1:]
        ious = compute_3d_iou(boxes[i], boxes[remaining])
        mask = ious < iou_threshold
        order = remaining[mask]
    return keep

Comparison Table

Common Challenges

1. Adverse Weather: Rain, fog, and snow degrade sensor performance differently requiring robust fusion
2. Long Tail Cases: Rare objects and unusual scenarios have limited training data
3. Real-Time Constraints: Full perception pipeline must execute within 50ms latency budget
4. Calibration Drift: Sensor mounting changes over time require online calibration
5. Occlusion Handling: Partial visibility requires reasoning about occluded objects

Case Study: nuScenes Benchmark

nuScenes contains 1000 driving scenes from Boston and Singapore with 1.4M LiDAR sweeps and 1.4M camera images. BEVFusion achieves 72.8% NDS by fusing camera features lifted to BEV with LiDAR voxel features using cross-attention. CenterPoint reaches 65.4% NDS with LiDAR-only using center-based heatmap detection and velocity regression. The dataset includes challenging scenarios: 20% scenes in rain, 10% at night, and 5% with construction zones. The ego vehicle travels at 5-58 km/h with up to 324 tracked objects per frame. Training uses AdamW with learning rate 0.002, batch size 4 across 8 A100 GPUs for 20 epochs.

Key Takeaways

• Multi-sensor fusion combines camera semantics with LiDAR geometry for robust perception
• Bird's Eye View transformation provides a unified representation for cross-modal fusion
• Kalman filtering enables robust tracking with velocity estimation from position measurements
• Center-based detection avoids anchor design while handling variable object sizes
• Real-time performance requires efficient backbones and parallel processing pipelines
• Adverse weather and lighting conditions demand modality-specific robustness
• End-to-end learning approaches are emerging but modular systems remain dominant in production