Latent Diffusion Models: Architecture and Theory

Module: Generative AI | Difficulty: Advanced

Latent Diffusion Objective

where is the VAE encoder output.

KL-f8 Autoencoder

Compresses by factor :

Cross-Attention Conditioning

where , ,

import torch, torch.nn as nn

class LatentDiffusion(nn.Module):
    def __init__(self, unet, vae, clip_model):
        super().__init__()
        self.unet = unet
        self.vae = vae
        self.clip = clip_model
    def forward(self, x, t, text):
        z = self.vae.encode(x)
        cond = self.clip.encode_text(text)
        noise = torch.randn_like(z)
        z_t = self.q_sample(z, t, noise)
        return ((noise - self.unet(z_t, t, cond))**2).mean()

Research Insight: Latent diffusion is 16-64x more efficient than pixel diffusion because the latent space has lower dimensionality while preserving perceptual quality.