Video Classification, Temporal Modeling, and Video QA

Module: Computer Vision | Difficulty: Premium

Temporal Convolution for Video

where is the dilation factor.

SlowFast Temporal Ratio

Video QA Formulation

where selects frames uniformly.

| Model | Kinetics-400 | SSv2 | VideoQA | Approach | |-------|-------------|------|---------|----------| | TSM | 74.1% | 35.9% | - | Temporal shift | | SlowFast | 79.8% | 52.6% | - | Dual pathway | | TimeSformer | 80.7% | 59.3% | - | Divided attention | | InternVideo2 | 92.0% | 75.8% | 71.2% | Large-scale |

import torch
import torch.nn as nn
import torch.nn.functional as F

class TemporalShiftModule(nn.Module):
    def __init__(self, channels, num_frames, shift_div=8):
        super().__init__()
        self.channels = channels
        self.num_frames = num_frames
        self.shift_div = shift_div
        self.fold_div = channels // shift_div

    def forward(self, x):
        N, T, C, H, W = x.shape
        out = x.clone()
        c_per = C // self.fold_div
        out[:, 1:, :c_per] = x[:, :-1, :c_per]
        out[:, :-1, c_per:2*c_per] = x[:, 1:, c_per:2*c_per]
        return out

class VideoTransformer(nn.Module):
    def __init__(self, num_frames=8, num_classes=400):
        super().__init__()
        self.temporal_embed = nn.Parameter(
            torch.randn(1, num_frames, 1, 1) * 0.02)
        self.spatial_embed = nn.Parameter(
            torch.randn(1, 1, 14, 14) * 0.02)
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=768, nhead=12, batch_first=True)
        self.transformer = nn.TransformerEncoder(
            encoder_layer, num_layers=12)
        self.head = nn.Linear(768, num_classes)

    def forward(self, patch_features):
        B, T, N, D = patch_features.shape
        patch_features = (patch_features
                          + self.temporal_embed
                          + self.spatial_embed)
        patch_features = patch_features.reshape(B * T, N, D)
        features = self.transformer(patch_features)
        features = features.mean(dim=1)
        features = features.reshape(B, T, -1).mean(dim=1)
        return self.head(features)

def sample_frames(video, num_frames=8):
    T_total = video.shape[0]
    indices = torch.linspace(0, T_total - 1, num_frames).long()
    return video[indices]

Research Insight: Video understanding is transitioning from clip-level to long-form analysis. InternVideo2 demonstrated that large-scale pretraining on diverse video data (including web videos, instructional content, and social media) produces representations that transfer across a wide range of video tasks. The key challenge is efficiently processing videos with thousands of frames while maintaining temporal resolution. Adaptive token selection and sparse attention are promising directions for scaling video transformers to longer sequences.