Score Distillation Sampling

Module: Generative AI | Difficulty: Advanced

SDS Loss

where is the differentiable renderer output.

Mode Seeking Problem

SDS tends to produce oversaturated, burnt images because it optimizes for a single noise realization.

Solutions

1. Prolific Dreamer (VSD): Uses VSD loss with LoRA fine-tuned diffusion
2. Magic3D: Coarse-to-fine optimization
3. DreamCraft3D: Multi-view consistency + SDS

def sds_loss(model, renderer, camera_params, text_cond, t):
    with torch.no_grad():
        image = renderer(camera_params)
        image = vae.encode(image)
        noise = torch.randn_like(image)
        xt = scheduler.add_noise(image, noise, t)
        eps_pred = model(xt, t, text_cond)
    grad = eps_pred - noise
    return (grad * xt).mean()

Research Insight: SDS is equivalent to minimizing the KL divergence between the rendered image distribution and the conditional diffusion distribution. The burning effect comes from the high variance of the gradient estimator.