Depth Estimation, Stereo Matching, and 3D Reconstruction

Module: Computer Vision | Difficulty: Premium

Disparity-Depth Relationship

where is focal length, is baseline, is disparity.

Epipolar Constraint

where is the fundamental matrix and are corresponding points in homogeneous coordinates.

Census Transform

where if , else .

Semi-Global Matching Cost

import torch
import torch.nn as nn

class CostVolume(nn.Module):
    def __init__(self, max_disp=192):
        super().__init__()
        self.max_disp = max_disp

    def forward(self, feat_l, feat_r):
        N, C, H, W = feat_l.shape
        cost = torch.zeros(N, C, self.max_disp, H, W, device=feat_l.device)
        for d in range(self.max_disp):
            if d > 0:
                cost[:, :, d, :, d:] = feat_l[:, :, :, d:] * feat_r[:, :, :, :-d]
            else:
                cost[:, :, d, :, :] = feat_l * feat_r
        return cost / C

def compute_disparity(cost, return_argmax=True):
    if return_argmax:
        return cost.mean(dim=1).argmax(dim=1).float()
    prob = torch.softmax(-cost.mean(dim=1) * 10, dim=1)
    disp_range = torch.arange(cost.shape[2], device=cost.device).float()
    return (prob * disp_range[None, :, None, None]).sum(dim=1)

Research Insight: Modern stereo methods have converged on learning-based cost volume construction with iterative refinement. CREStereo introduced cascaded cost volumes at multiple resolutions, enabling accurate matching for both textureless regions and fine structures. The key remaining challenge is handling occlusion boundaries, where left-right consistency checks and visibility masks are essential. Self-supervised stereo training (using photometric consistency as loss) has enabled large-scale pretraining without depth annotations.