Consistency Distillation

Module: Generative AI | Difficulty: Advanced

Consistency Distillation

Train student to match teacher's ODE trajectory:

Progressive Distillation

1. Train teacher with steps
2. Train student in 2 steps
3. Student becomes new teacher
4. Repeat times

One-Step Generation

class ConsistencyDistillation:
    def __init__(self, teacher, student, ema_decay=0.999):
        self.teacher = teacher
        self.student = student
        self.teacher_ema = EMA(teacher, ema_decay)
    def loss(self, x0, t1, t2):
        xt1 = self.add_noise(x0, t1)
        xt2 = self.add_noise(x0, t2)
        s1 = self.student(xt1, t1)
        with torch.no_grad():
            s2 = self.teacher_ema(xt2, t2)
        return ((s1 - s2)**2).mean()