Video Diffusion Models

Module: Generative AI | Difficulty: Advanced

Video Diffusion

Temporal Attention

Applied across frames after spatial attention.

Frame Interpolation

import torch, torch.nn as nn

class VideoDiffusion(nn.Module):
    def __init__(self, spatial_attn, temporal_attn):
        super().__init__()
        self.spatial = spatial_attn
        self.temporal = temporal_attn
    def forward(self, x, t):
        B, T = x.shape[:2]
        x = x.view(B*T, *x.shape[2:])
        x = self.spatial(x, t.repeat_interleave(T))
        x = x.view(B, T, *x.shape[1:])
        x = self.temporal(x)
        return x

Research Insight: Video diffusion models struggle with temporal consistency. Key solutions: temporal attention layers, optical flow guidance, and frame-by-frame refinement with global coherence.