Monocular Depth Estimation

Monocular Depth Estimation Overview

Monocular depth estimation predicts a dense depth map from a single RGB image, estimating the distance of each pixel from the camera. This ill-posed problem requires the model to infer depth from monocular cues such as perspective, texture gradients, object sizes, and occlusion. Modern approaches use encoder-decoder architectures with skip connections and multi-scale losses, while self-supervised methods train on stereo or monocular video without ground truth depth.

Theory: Encoder-Decoder Depth Prediction

The encoder-decoder framework processes the input image through a pretrained backbone that extracts multi-scale features capturing semantic and spatial information. The decoder progressively upsamples these features while integrating skip connections from the encoder to recover spatial details lost during downsampling. Multi-scale prediction heads at different decoder levels enable the network to capture both local details and global context.

Self-supervised depth estimation removes the need for expensive ground truth by leveraging photometric consistency between consecutive video frames or stereo pairs. The network predicts depth and ego-motion, then synthesizes one view from another using differentiable warping. Training minimizes the photometric reconstruction error between the synthesized and real images.

Scale ambiguity is a fundamental challenge in monocular depth since the model cannot determine absolute scale without additional information. Techniques like scale normalization, invariant loss, and training with known camera intrinsics help resolve this ambiguity.

Mathematical Foundations

The scale-invariant logarithmic depth loss:

Where each parameter means:

• is the log-depth error at pixel
• is the predicted depth at pixel
• is the ground truth depth at pixel
• is the number of valid pixels
• is the scale regularization weight (typically 0.85)

The photometric reconstruction loss for self-supervised training:

Where each parameter means:

• is the target image at time
• is the source image warped to the target view
• is the robust L1 loss or SSIM similarity measure
• The summation is over all valid pixels with visible depth
• Photometric consistency assumes static scene and known camera motion

The multi-scale gradient matching loss:

Where each parameter means:

• is the predicted depth at pixel
• is the ground truth depth at pixel
• and are spatial gradients in horizontal and vertical directions
• The loss encourages predicted depth edges to align with ground truth edges

Architecture Design

Implementation

import torch
import torch.nn as nn
import torchvision.models as models


class DepthEstimationNet(nn.Module):
    def __init__(self):
        super(DepthEstimationNet, self).__init__()
        encoder = models.densenet161(pretrained=True)
        self.encoder_blocks = nn.ModuleList([
            nn.Sequential(encoder.features[:4]),
            nn.Sequential(encoder.features[4:6]),
            nn.Sequential(encoder.features[6:8]),
            nn.Sequential(encoder.features[8:]),
        ])
        self.decoder_blocks = nn.ModuleList([
            self._make_decoder(2208, 512),
            self._make_decoder(512 + 512, 256),
            self._make_decoder(256 + 256, 128),
            self._make_decoder(128 + 64, 64),
        ])
        self.final_conv = nn.Conv2d(64, 1, kernel_size=3, padding=1)
        self.sigmoid = nn.Sigmoid()

    def _make_decoder(self, in_ch, out_ch):
        return nn.Sequential(
            nn.ConvTranspose2d(in_ch, out_ch, 3, stride=2, padding=1, output_padding=1),
            nn.BatchNorm2d(out_ch), nn.ReLU(inplace=True),
            nn.Conv2d(out_ch, out_ch, 3, padding=1),
            nn.BatchNorm2d(out_ch), nn.ReLU(inplace=True)
        )

    def forward(self, x):
        skips = []
        for i, block in enumerate(self.encoder_blocks):
            x = block(x)
            if i < 3:
                skips.append(x)
        for i, decoder in enumerate(self.decoder_blocks):
            if i > 0:
                x = torch.cat([x, skips[-i]], dim=1)
            x = decoder(x)
        depth = self.sigmoid(self.final_conv(x))
        return depth


def silog_loss(pred, target, lambda_weight=0.85):
    valid = target > 0
    pred = pred[valid]
    target = target[valid]
    log_diff = torch.log(pred) - torch.log(target)
    silog = torch.mean(log_diff ** 2) - lambda_weight * (torch.mean(log_diff)) ** 2
    return torch.sqrt(silog)

Comparison Table

Common Challenges

1. Scale Ambiguity: Monocular images lack absolute scale information requiring training priors or known camera parameters
2. Texture-less Regions: Uniform surfaces provide no visual cues for depth estimation
3. Reflective Surfaces: Mirrors and glossy surfaces create incorrect depth predictions
4. Far Objects: Depth uncertainty increases quadratically with distance
5. Domain Gap: Models trained on one dataset may fail on different camera setups or environments

Self-Supervised Depth from Stereo

Self-supervised depth estimation uses stereo image pairs during training by warping one view to the other using predicted depth and known camera geometry. The photometric reconstruction loss compares the warped source image with the target image, providing supervision without ground truth depth. Auto-masking excludes static pixels where the assumption of a static scene fails, such as moving objects or textureless regions.

The minimum reprojection loss takes the per-pixel minimum across multiple source views, handling occlusions where some views do not see the same point. SSIM-based photometric loss combined with L1 provides robust similarity measurement that accounts for illumination changes. Left-right consistency checking enforces that depth from left-to-right warping matches right-to-left warping, improving depth accuracy at object boundaries.

Training schedules for self-supervised depth typically use 10-20 epochs with learning rate warmup over 4000 iterations and exponential decay. The auto-masking threshold of 0.5 prevents training on pixels where the reconstructed image is less similar than the original, which occurs for static regions or moving objects.

Uncertainty Estimation

Depth prediction uncertainty quantifies the reliability of predicted depths, crucial for safety-critical applications. Aleatoric uncertainty from sensor noise is estimated by predicting both depth and variance, while epistemic uncertainty from model uncertainty requires Monte Carlo dropout or ensemble methods. The uncertainty-aware training loss weights the photometric reconstruction by the predicted uncertainty, allowing the model to be less penalized for uncertain regions.

Depth completion fills in sparse LiDAR depth maps using guidance from RGB images. The network takes sparse depth and high-resolution RGB as input and produces dense depth through learned interpolation. Edge-aware depth completion preserves depth discontinuities using bilateral guidance from the RGB image.

Case Study: KITTI Benchmark

KITTI contains 93K stereo pairs from a driving platform with LiDAR ground truth at 0.1 to 80m range. MiDaS v3.1 achieves 3.972 RMSE and 0.110 REL using ViT-Large encoder pretrained on 12 datasets. Self-supervised MonoDepth2 reaches 4.831 RMSE without ground truth by training on monocular video with photometric consistency and auto-masking for static pixels. AdaBins achieves 3.520 RMSE by predicting adaptive bin centers for depth discretization, combining regression and classification losses. Training uses AdamW optimizer with cosine annealing schedule over 20 epochs. The depth accuracy degrades quadratically with distance: at 20m the median REL is 5%, while at 80m it increases to 15%, highlighting the fundamental limitation of monocular depth estimation at long range.

Practical Deployment Considerations

Real-time depth estimation on embedded platforms requires model optimization through quantization, pruning, and knowledge distillation. INT8 quantization reduces model size by 4x with less than 1% accuracy degradation on modern GPUs. TensorRT optimization achieves 30 FPS on Jetson AGX Xavier for 640x480 input resolution.

Depth post-processing removes outliers using bilateral filtering, hole filling through morphological operations, and temporal filtering across video frames. These operations smooth depth maps while preserving edges, improving visual quality and downstream task performance. The combination of guided filtering with RGB images produces sharp depth boundaries aligned with object edges.

Depth for Downstream Applications

Estimated depth maps enable numerous downstream applications including 3D object detection, scene understanding, and augmented reality. Depth-guided object detection uses depth to define 3D proposals and filter distant objects, improving detection accuracy by 5-8% on distant objects. The depth information enables scale-aware detection that correctly sizes objects regardless of camera distance.

3D scene reconstruction combines depth maps with camera poses to build textured meshes. The depth map provides per-pixel 3D coordinates, while the camera pose transforms points into a common reference frame. Multi-view stereo approaches fuse depth from multiple viewpoints to produce watertight meshes suitable for visualization and interaction.

Simultaneous localization and mapping (SLAM) uses depth for camera tracking and map building. The depth provides scale information that monocular SLAM lacks, enabling metric-scale trajectory estimation. Dense SLAM systems like KinectFusion use depth maps to build detailed 3D maps in real-time for robot navigation.

Depth Prediction from Monocular Video

Self-supervised monocular depth estimation trains on single video sequences by enforcing photometric consistency between consecutive frames. The network predicts depth and ego-motion simultaneously, then warps one frame to another using the predicted depth and camera motion. This approach eliminates the need for expensive LiDAR ground truth while achieving competitive accuracy.

The training loss combines photometric reconstruction with depth smoothness regularization. The smoothness term penalizes depth gradients that do not correspond to image gradients, encouraging sharp depth edges aligned with object boundaries. Multi-scale evaluation computes the photometric loss at multiple resolutions to handle large motions and textureless regions.

Temporal consistency losses enforce smooth depth predictions across frames, reducing flickering in video depth estimation. The forward-backward consistency check verifies that depth from frame t to t+1 matches depth from t+1 to t, providing self-supervision for depth accuracy. These techniques reduce temporal jitter by 40% compared to frame-by-frame prediction.

Depth in Robotics and Navigation

Depth estimation enables autonomous navigation by providing distance information for obstacle avoidance. The robot uses depth maps to identify free space, plan collision-free paths, and estimate traversability of terrain. Real-time depth at 30 FPS on mobile platforms requires efficient architectures optimized for edge deployment.

Simultaneous localization and mapping (SLAM) systems use depth to build 3D maps while tracking camera pose. Dense SLAM reconstructs detailed environments by fusing depth maps across viewpoints, enabling applications in augmented reality and virtual tour creation. The depth accuracy directly impacts map quality and localization precision.

Grasp planning for robotic manipulation uses depth to determine optimal gripper poses for picking objects. The depth map provides 3D point clouds of objects, enabling calculation of grasp poses that avoid collisions and maximize stability. Depth-based grasp planning achieves 90% success rates on standard objects compared to 70% with 2D-only approaches.

Depth from Focus and Defocus

Depth from focus techniques estimate depth by analyzing the sharpness of image regions across multiple focal settings. Objects at the focal distance appear sharp while others are blurred. By capturing images at different focus distances and measuring sharpness at each pixel, the depth corresponding to maximum sharpness provides the depth estimate.

Depth from defocus uses the blur amount to infer depth since blur radius is related to the distance from the focal plane. The circle of confusion model relates blur radius to depth through camera intrinsics and focus settings. This approach requires calibrated cameras with known optical parameters but provides dense depth from just 2-3 images.

The defocus cue is particularly effective for macro photography and microscopy where depth of field is shallow. Combining defocus with stereo provides complementary depth information: stereo works well for distant objects while defocus excels at close range. Multi-cue fusion leverages the strengths of each approach across the depth range.

Key Takeaways

• Encoder-decoder architectures with skip connections effectively recover spatial details for dense prediction
• Multi-scale losses provide gradients at different resolutions enabling better edge preservation
• Self-supervised training eliminates expensive ground truth collection using photometric consistency
• ViT-based encoders capture long-range dependencies superior to CNN backbones
• Scale-invariant losses handle the inherent ambiguity of monocular depth estimation
• Multi-dataset training improves generalization across different environments and camera setups
• Edge-aware losses improve depth boundary sharpness for downstream applications