GAN Training Dynamics: A Dynamical Systems Perspective

Module: Generative AI | Difficulty: Advanced

Nash Equilibrium

Theorem (Goodfellow, 2014): Global minimum of is when .

Mode Collapse as Bifurcation

When becomes too strong, the gradient dynamics undergo a saddle-node bifurcation, creating stable fixed points corresponding to mode collapse.

Spectral Normalization (Miyato, 2018)

Gradient Penalty (Gulrajani, 2017)

def spectral_norm(weight):
    u = torch.randn(weight.size(0), 1)
    for _ in range(5):
        v = weight.t() @ u; v = v / (v.norm() + 1e-8)
        u = weight @ v; u = u / (u.norm() + 1e-8)
    sigma = (u.t() @ weight @ v).item()
    return weight / sigma

| Method | FID | Stability | |--------|-----|-----------| | WGAN-GP | 31.2 | High | | Spectral Norm | 28.7 | High | | SN + R1 | 22.1 | High |

Research Insight: Two-timescale update rules (D updated faster than G) are critical for convergence. The convergence rate is .