Noise Scheduling in Diffusion Models

Module: Generative AI | Difficulty: Advanced

Linear Schedule

Cosine Schedule (Nichol & Dhariwal, 2021)

Shifted Schedule

Impact on Quality

| Schedule | FID | NLL | |----------|-----|-----| | Linear | 3.17 | -1.18 | | Cosine | 2.97 | -1.21 | | Shifted | 2.83 | -1.24 | | Learned | 2.78 | -1.26 |

def cosine_schedule(T, s=0.008):
    steps = torch.arange(T+1, dtype=torch.float64) / T
    alphas = torch.cos((steps + s) / (1 + s) * math.pi / 2) ** 2
    alphas = alphas / alphas[0]
    betas = 1 - (alphas[1:] / alphas[:-1])
    return torch.clamp(betas, 0.0001, 0.9999)

Research Insight: The noise schedule determines which frequencies of the data are learned first. Linear schedules learn low frequencies first; cosine schedules learn all frequencies more uniformly, leading to better quality.