Video Understanding and Temporal Action Localization

Module: Computer Vision | Difficulty: Premium

Temporal Segment Network (TSN)

where is the feature from segment , denotes fusion (e.g., average pooling).

SlowFast Networks

• Slow pathway: low frame rate, captures spatial semantics ( frames)
• Fast pathway: high frame rate, captures motion ( frames, )

VideoMAE Pretraining

Reconstruct masked video patches:

import torch
import torch.nn as nn

class TemporalShiftModule(nn.Module):
    def __init__(self, channels, num_frames, shift_div=8):
        super().__init__()
        self.channels = channels
        self.num_frames = num_frames
        self.shift_div = shift_div
        self.fold_div = channels // shift_div

    def forward(self, x):
        N, T, C, H, W = x.shape
        out = x.clone()
        c_per = C // self.fold_div
        out[:, 1:, :c_per] = x[:, :-1, :c_per]
        out[:, :-1, c_per:2*c_per] = x[:, 1:, c_per:2*c_per]
        return out

class VideoClassifier(nn.Module):
    def __init__(self, num_classes=400, num_frames=8):
        super().__init__()
        import torchvision.models as models
        self.backbone = models.resnet50(pretrained=True)
        self.backbone.fc = nn.Linear(2048, num_classes)
        self.temporal_pool = nn.AdaptiveAvgPool1d(1)
        self.num_frames = num_frames

    def forward(self, x):
        N, T, C, H, W = x.shape
        features = []
        for t in range(T):
            feat = self.backbone(x[:, t])
            features.append(feat)
        features = torch.stack(features, dim=1)
        pooled = self.temporal_pool(features.squeeze(-1)).squeeze(-1)
        return self.backbone.fc(pooled)

Research Insight: Video understanding has been revolutionized by masked video modeling (VideoMAE), which learns rich spatiotemporal representations from unlabeled video. By masking 90%+ of video patches and reconstructing them, the model learns fine-grained motion patterns without manual labels. The combination of large-scale pretraining (InternVideo2) with task-specific fine-tuning achieves remarkable transfer across action recognition, temporal localization, and video QA tasks.