Score-Based Diffusion: Mathematical Foundations

Module: Generative AI | Difficulty: Advanced

The score function is the gradient of the log-density. Unlike , the score function can be estimated without computing the intractable normalizing constant.

Score Matching Objective (Hyvarinen, 2005)

Theorem: This is minimized when .

Proof: Expanding and integrating by parts:

Denoising Score Matching (Vincent, 2011)

Langevin Dynamics

Score-Based SDE (Song et al., 2021)

Forward:

Reverse:

import torch, torch.nn as nn

class ScoreNetwork(nn.Module):
    def __init__(self, dim, hidden=256):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(dim + 1, hidden), nn.SiLU(),
            nn.Linear(hidden, hidden), nn.SiLU(),
            nn.Linear(hidden, hidden), nn.SiLU(),
            nn.Linear(hidden, dim))
    def forward(self, x, sigma):
        return self.net(torch.cat([x, sigma], dim=-1))

def dsm_loss(model, x, sigma=0.1):
    noise = torch.randn_like(x) * sigma
    return 0.5 * ((model(x + noise, sigma.expand(x.size(0),1)) + noise/sigma**2)**2).mean()

Research Insight: Score-based models outperform GANs because the denoising objective provides gradient signal everywhere in data space, unlike GANs which only learn the support.