Diffusion Transformers & Video Generation
1. Diffusion Models Overview
1.1 Forward Process
The forward diffusion process adds Gaussian noise to data over timesteps:
where is the noise schedule. Using the reparameterization trick:
where and .
1.2 Reverse Process
The reverse process learns to denoise:
where the mean is typically parameterized as:
1.3 Training Objective
The simplified denoising objective:
where and .
2. Diffusion Transformers (DiT)
2.1 Architecture Overview
2.2 Adaptive Layer Norm (AdaLN-Zero)
The key innovation in DiT is Adaptive Layer Normalization, which conditions the network on the timestep and optional class label :
where and are generated from the timestep embedding:
The AdaLN-Zero variant initializes all projection layers to zero, ensuring the transformer initially behaves as an identity function:
2.3 DiT Block Computation
class DiTBlock(nn.Module):
def __init__(self, d_model, n_heads):
super().__init__()
self.norm1 = LayerNorm(d_model)
self.attn = MultiHeadAttention(d_model, n_heads)
self.norm2 = LayerNorm(d_model)
self.ffn = FeedForward(d_model)
self.adaLN_modulation = nn.Linear(d_model, 6 * d_model)
def forward(self, x, t_emb):
# AdaLN modulation
shift_msa, scale_msa, gate_msa, \
shift_mlp, scale_mlp, gate_mlp = self.adaLN_modulation(t_emb).chunk(6)
# Self-attention with AdaLN
x = x + gate_msa * self.attn(
modulate(self.norm1(x), shift_msa, scale_msa)
)
# FFN with AdaLN
x = x + gate_mlp * self.ffn(
modulate(self.norm2(x), shift_mlp, scale_mlp)
)
return x
def modulate(x, shift, scale):
return x * (1 + scale) + shift
3. Latent Diffusion Models (LDM)
3.1 Architecture
Latent Diffusion Models (Rombach et al., 2022) perform diffusion in a compressed latent space rather than pixel space:
where is encoded into latent space by the encoder .
3.2 VAE Encoder/Decoder
Encoder:
Decoder:
with downsampling factor (typically ).
VAE Training Objective:
3.3 Latent Diffusion Pipeline
4. Video Diffusion Models
4.1 Temporal Extension
Video extends images with temporal dimension: where is the number of frames.
Temporal attention is added to the spatial DiT:
4.2 Temporal Attention
For a video with frames, each with patches:
where indices span spatial patches and span temporal frames.
4.3 Causal Temporal Masking
For autoregressive video generation:
This ensures frame can only attend to frames .
4.4 Video Diffusion Training
where represents the noised video sequence.
5. Sora-Style Architectures
5.1 Key Design Principles
Sora (OpenAI, 2024) introduced several innovations:
- Spacetime patches: Patchify video as volumetric patches
- Diffusion Transformer: DiT backbone operating on patch tokens
- Variable resolution: Support different aspect ratios and durations
- Conditioning: Text prompts, images, or other videos
5.2 Patchification
Video is divided into patches:
Total number of patches:
5.3 Scaling Laws for Video
Computational requirements scale with:
where is the number of patches, is the model dimension, and is the number of transformer layers.
6. Classifier-Free Guidance
6.1 Formulation
Classifier-free guidance (Ho & Salimans, 2022) combines conditional and unconditional generation:
where:
- : unconditional prediction (null condition)
- : conditional prediction
- : guidance scale
6.2 Effect of Guidance Scale
| Effect | |
|---|---|
| 1.0 | No guidance (standard conditional) |
| 2.0-3.0 | Moderate guidance (good quality-diversity balance) |
| 5.0-8.0 | Strong guidance (high quality, low diversity) |
| >10.0 | Over-guided (artifacts, saturation) |
6.3 Negative Prompt Guidance
where is the positive prompt and is the negative prompt.
7. Sampling Strategies
7.1 DDPM Sampling
7.2 DDIM Sampling
Deterministic sampling with fewer steps:
7.3 DPM-Solver
High-order ODE solver for diffusion:
Achieves good quality with 10-20 steps.
8. Stable Diffusion Implementation
from diffusers import StableDiffusionPipeline
import torch
# Load pre-trained Stable Diffusion
pipe = StableDiffusionPipeline.from_pretrained(
"stabilityai/stable-diffusion-2-1",
torch_dtype=torch.float16
).to("cuda")
# Generate image
prompt = "A futuristic city at sunset, cyberpunk style"
image = pipe(
prompt=prompt,
negative_prompt = "blurry, low quality, distorted",
num_inference_steps=50,
guidance_scale=7.5,
height=768,
width=1024,
).images[0]
# Custom DiT model
class VideoDiT(nn.Module):
def __init__(self, d_model=1024, n_heads=16, n_layers=24,
patch_size=(2, 8, 8), in_channels=4):
super().__init__()
self.patch_embed = PatchEmbed3D(patch_size, in_channels, d_model)
self.temporal_embed = nn.Embedding(max_frames, d_model)
self.pos_embed = nn.Parameter(torch.randn(1, max_patches, d_model))
self.blocks = nn.ModuleList([
DiTBlock(d_model, n_heads) for _ in range(n_layers)
])
self.final_norm = LayerNorm(d_model)
self.output_proj = nn.Linear(d_model, in_channels * prod(patch_size))
def forward(self, x, t, condition):
B, T, C, H, W = x.shape
x = self.patch_embed(x) # (B, N, D)
# Add positional and temporal embeddings
x = x + self.pos_embed + self.temporal_embed(
torch.arange(T, device=x.device)
).unsqueeze(0).expand(B, -1, -1).flatten(1, 2)
for block in self.blocks:
x = block(x, t, condition)
return self.output_proj(self.final_norm(x))
9. Evaluation Metrics
9.1 FID (Fréchet Inception Distance)
9.2 FVD (Fréchet Video Distance)
Extends FID to video using I3D features:
9.3 CLIP Score
10. References
- Peebles & Xie (2023). "Scalable Diffusion Models with Transformers." ICCV.
- Rombach et al. (2022). "High-Resolution Image Synthesis with Latent Diffusion Models." CVPR.
- Ho et al. (2022). "Video Diffusion Models." NeurIPS.
- Blattmann et al. (2023). "Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets." arXiv.
- OpenAI (2024). "Video generation models as world simulators." Technical Report.
- Esser et al. (2024). "Scaling Rectified Flow Transformers for High-Resolution Image Synthesis." ICML.