Video Understanding

Module: Computer Vision | Difficulty: Advanced

Video Representation

A video is a spatio-temporal volume .

Two-Stream Architecture

Spatial stream: RGB frames. Temporal stream: optical flow.

3D Convolution

Video Transformer (TimeSformer)

Temporal Action Localization

import torch
import torch.nn as nn

class I3D(nn.Module):
    def __init__(self, num_classes=400):
        super().__init__()
        self.conv3d = nn.Sequential(
            nn.Conv3d(3, 64, kernel_size=(3, 7, 7), stride=(1, 2, 2), padding=(1, 3, 3)),
            nn.BatchNorm3d(64),
            nn.ReLU(inplace=True),
            nn.MaxPool3d(kernel_size=(1, 3, 3), stride=(1, 2, 2), padding=(0, 1, 1)),
            nn.Conv3d(64, 128, kernel_size=(3, 3, 3), padding=(1, 1, 1)),
            nn.BatchNorm3d(128),
            nn.ReLU(inplace=True),
            nn.AdaptiveAvgPool3d((1, 1, 1)),
        )
        self.fc = nn.Linear(128, num_classes)
    
    def forward(self, x):
        x = self.conv3d(x)
        return self.fc(x.view(x.size(0), -1))

Key Takeaways

• Two-stream networks separate spatial and temporal information
• 3D convolutions capture spatio-temporal patterns directly
• Video transformers achieve SOTA on Kinetics and other benchmarks