Lucas-Kanade, Horn-Schunck, and Deep Optical Flow

Module: Computer Vision | Difficulty: Premium

Brightness Constancy Assumption

Taylor expansion yields the optical flow constraint:

Horn-Schunck Energy Functional

Lucas-Kanade (Local Method)

Minimize over a window :

End-Point Error (EPE)

| Method | Year | Sintel EPE | KITTI | Approach | |--------|------|------------|-------|----------| | Horn-Schunck | 1981 | 9.26 | 8.40 | Variational | | FlowNet2 | 2017 | 3.96 | 2.02 | CNN | | PWC-Net | 2018 | 3.45 | 1.75 | Pyramid | | RAFT | 2020 | 1.61 | 0.95 | Recurrent | | CRAFT | 2022 | 1.39 | 0.78 | Correlation |

import torch
import torch.nn as nn

class RAFTUpdateBlock(nn.Module):
    def __init__(self, hidden_dim=128):
        super().__init__()
        self.conv1 = nn.Conv2d(128 + 2 + 128, 128, 3, padding=1)
        self.conv2 = nn.Conv2d(128, 128, 3, padding=1)
        self.relu = nn.ReLU(inplace=True)
        self.flow_head = nn.Sequential(
            nn.Conv2d(128, 256, 3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(256, 2, 3, padding=1),
        )

    def forward(self, net, inp, corr):
        input = torch.cat([net, inp, corr], dim=1)
        net = self.relu(self.conv1(input))
        net = self.relu(self.conv2(net))
        delta_flow = self.flow_head(net)
        return net, delta_flow

def initialize_flow(img):
    N, C, H, W = img.shape
    flow = torch.zeros(N, 2, H, W, device=img.device)
    coords0 = torch.meshgrid(
        torch.arange(H, device=img.device),
        torch.arange(W, device=img.device), indexing='ij')
    coords0 = torch.stack(coords0, dim=0).float()
    return coords0.unsqueeze(0).repeat(N, 1, 1, 1), flow

Research Insight: RAFT (Recurrent All-Pairs Field Transforms) recast optical flow as an iterative refinement problem. By computing dense 4D correlation volumes and iteratively updating flow predictions using GRU cells, RAFT achieves state-of-the-art accuracy while being more generalizable across domains. The key insight is that optical flow estimation benefits from many small refinement steps rather than a single feedforward prediction, analogous to how iterative solvers outperform direct methods in linear algebra.