GANs for Image Generation

Module: Computer Vision | Difficulty: Advanced

The Minimax Game

Generative adversarial networks learn through a competitive game between two networks: the generator maps random noise to realistic images, while the discriminator classifies images as real or generated. This adversarial dynamic drives both networks to improve until the generator produces indistinguishable outputs.

The training objective is a minimax game:

Where each parameter means:

• — generator network mapping noise to images
• — discriminator network outputting probability of being real
• — true data distribution
• — prior noise distribution (typically )
• — discriminator's probability that is real
• — discriminator's probability that generated image is real
• Intuition: The discriminator maximizes its ability to distinguish real from fake, while the generator minimizes the discriminator's ability to do so; at equilibrium,

Wasserstein GAN

Standard GAN training suffers from mode collapse and vanishing gradients. WGAN replaces JS divergence with Wasserstein distance:

Where each parameter means:

• — set of 1-Lipschitz functions (enforced via weight clipping or gradient penalty)
• — critic (not discriminator) outputting unbounded real values
• Intuition: Wasserstein distance provides smooth gradients even when distributions don't overlap, solving the vanishing gradient problem of standard GANs

Spectral Normalization

Spectral normalization constrains the Lipschitz constant of the discriminator:

Where each parameter means:

• — weight matrix of layer
• — largest singular value
• — spectrally normalized weight matrix
• Intuition: By dividing each weight matrix by its largest singular value, we ensure the discriminator has Lipschitz constant 1, stabilizing training without weight clipping

Training Challenges

Mode Collapse

Mode collapse occurs when the generator produces limited variety, focusing on a few outputs that fool the discriminator:

Where each parameter means:

• — generator's output distribution
• — true data distribution
• Intuition: The generator finds a few "safe" outputs that consistently fool the discriminator and stops exploring, resulting in limited diversity

Evaluation Metrics

The Fréchet Inception Distance measures quality and diversity:

Where each parameter means:

• — mean and covariance of real image features
• — mean and covariance of generated image features
• Features extracted from Inception-v3 pool3 layer
• Intuition: FID measures the distance between feature distributions; lower FID means generated images are more similar to real images in both quality and diversity

GAN Architecture Comparison

Complete DCGAN Training Loop

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader


class Generator(nn.Module):
    def __init__(self, z_dim=100, channels=3):
        super().__init__()
        self.net = nn.Sequential(
            nn.ConvTranspose2d(z_dim, 512, 4, 1, 0, bias=False),
            nn.BatchNorm2d(512), nn.ReLU(True),
            nn.ConvTranspose2d(512, 256, 4, 2, 1, bias=False),
            nn.BatchNorm2d(256), nn.ReLU(True),
            nn.ConvTranspose2d(256, 128, 4, 2, 1, bias=False),
            nn.BatchNorm2d(128), nn.ReLU(True),
            nn.ConvTranspose2d(128, 64, 4, 2, 1, bias=False),
            nn.BatchNorm2d(64), nn.ReLU(True),
            nn.ConvTranspose2d(64, channels, 4, 2, 1),
            nn.Tanh()
        )

    def forward(self, z):
        return self.net(z.view(-1, z.size(1), 1, 1))


class Discriminator(nn.Module):
    def __init__(self, channels=3):
        super().__init__()
        self.net = nn.Sequential(
            nn.Conv2d(channels, 64, 4, 2, 1),
            nn.LeakyReLU(0.2, True),
            nn.Conv2d(64, 128, 4, 2, 1),
            nn.BatchNorm2d(128), nn.LeakyReLU(0.2, True),
            nn.Conv2d(128, 256, 4, 2, 1),
            nn.BatchNorm2d(256), nn.LeakyReLU(0.2, True),
            nn.Conv2d(256, 512, 4, 2, 1),
            nn.BatchNorm2d(512), nn.LeakyReLU(0.2, True),
            nn.Conv2d(512, 1, 4, 1, 0),
            nn.Sigmoid()
        )

    def forward(self, x):
        return self.net(x).view(-1)


def train_dcgan(epochs=50, batch_size=128, lr=0.0002, z_dim=100):
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    transform = transforms.Compose([
        transforms.Resize(64), transforms.CenterCrop(64),
        transforms.ToTensor(), transforms.Normalize([0.5]*3, [0.5]*3)
    ])
    dataset = datasets.CIFAR10(root='./data', train=True,
                               download=True, transform=transform)
    loader = DataLoader(dataset, batch_size=batch_size, shuffle=True)
    G = Generator(z_dim).to(device)
    D = Discriminator().to(device)
    opt_G = optim.Adam(G.parameters(), lr=lr, betas=(0.5, 0.999))
    opt_D = optim.Adam(D.parameters(), lr=lr, betas=(0.5, 0.999))
    criterion = nn.BCELoss()
    for epoch in range(epochs):
        for real, _ in loader:
            real = real.to(device)
            batch = real.size(0)
            z = torch.randn(batch, z_dim, device=device)
            fake = G(z).detach()
            D_real = D(real)
            D_fake = D(fake)
            D_loss = criterion(D_real, torch.ones(batch, device=device)) + \
                     criterion(D_fake, torch.zeros(batch, device=device))
            opt_D.zero_grad()
            D_loss.backward()
            opt_D.step()
            z = torch.randn(batch, z_dim, device=device)
            fake = G(z)
            D_fake = D(fake)
            G_loss = criterion(D_fake, torch.ones(batch, device=device))
            opt_G.zero_grad()
            G_loss.backward()
            opt_G.step()
    return G, D


G, D = train_dcgan(epochs=10)
print(f"Generator params: {sum(p.numel() for p in G.parameters()):,}")
print(f"Discriminator params: {sum(p.numel() for p in D.parameters()):,}")

Common Challenges

1. Mode Collapse: Generator produces limited variety, focusing on outputs that fool the discriminator
2. Training Instability: G and D must maintain careful balance; too strong D kills G gradients, too weak D allows fakes
3. Evaluation Difficulty: FID and IS are imperfect metrics; visual quality assessment remains subjective
4. Computational Cost: Training GANs requires significant GPU resources and careful hyperparameter tuning
5. Artifact Generation: GANs may produce artifacts like checkerboard patterns or unrealistic textures

Case Study: Face Generation

NVIDIA's StyleGAN2 (2020) generates photorealistic faces at 1024×1024 resolution with FID 2.84 on FFHQ dataset (70,000 faces). The model uses a mapping network to transform a 512-dim latent code into style parameters that control each layer through adaptive instance normalization. Training required 8 GPUs for 2 weeks on the FFHQ dataset. The key innovation was weight demodulation, which eliminates water droplet artifacts caused by the original AdaIN approach. A latent space analysis revealed that semantic attributes (age, gender, expression) are distributed smoothly, enabling meaningful interpolation and attribute manipulation. The system has been used in over 500 research papers and commercial applications.

Advanced GAN Architectures

StyleGAN Architecture

StyleGAN introduces a mapping network that transforms the latent code into an intermediate space , which then controls the generator through adaptive instance normalization (AdaIN):

Where each parameter means:

• — feature map at layer
• — style scale and bias computed from
• — channel-wise mean and standard deviation
• Intuition: By injecting style information at each layer through affine transformations, the generator can control different aspects of the image at different scales (coarse: pose, shape; fine: colors, textures)

StyleGAN2 Weight Demodulation

StyleGAN2 replaces AdaIN with weight demodulation to eliminate artifacts:

Where each parameter means:

• — demodulated weight tensor
• — style scale for channel
• — original weight tensor
• Intuition: Weight demodulation normalizes weights per-output-channel instead of per-sample, eliminating the "water droplet" artifacts caused by AdaIN

Progressive Growing

Progressive growing trains the generator and discriminator by gradually adding layers:

Where each parameter means:

• — generator producing images
• Training progresses from low to high resolution
• Intuition: By starting with low-resolution images and gradually increasing resolution, the model learns coarse structure before fine details, stabilizing training

GAN Training Techniques

Two-Timescale Update Rule (TTUR)

TTUR uses different learning rates for generator and discriminator:

Where each parameter means:

• — discriminator learning rate (typically 4x larger)
• — generator learning rate (typically 0.1-0.4)
• Intuition: The discriminator should be updated faster than the generator to provide stable gradient signals; typical ratios are 4:1

Gradient Penalty

WGAN-GP enforces Lipschitz constraint through gradient penalty:

Where each parameter means:

• — random interpolation between real and fake samples
• — penalty coefficient (typically 10)
• Intuition: By penalizing the gradient norm of the discriminator to be close to 1, we enforce the Lipschitz constraint more smoothly than weight clipping

GAN Evaluation Metrics

Fréchet Inception Distance (FID)

FID measures the distance between real and generated image feature distributions:

Where each parameter means:

• — mean and covariance of real image features
• — mean and covariance of generated image features
• Features extracted from Inception-v3 pool3 layer
• Intuition: FID measures the distance between feature distributions; lower FID means generated images are more similar to real images in both quality and diversity

Inception Score (IS)

IS measures quality and diversity of generated images:

Where each parameter means:

• — conditional label distribution (class probabilities given image )
• — marginal label distribution (average class probabilities)
• Intuition: High IS means images are both clearly classifiable (low conditional entropy) and diverse (high marginal entropy); however, IS has limitations as it doesn't compare to real data

Key Takeaways

• GANs learn through adversarial competition between generator and discriminator
• Wasserstein distance provides stable gradients and better training dynamics
• Spectral normalization constrains discriminator Lipschitz constant without weight clipping
• FID is the standard metric measuring both quality and diversity of generated images
• Mode collapse and training instability remain key challenges in GAN training
• Style-based architectures enable controllable generation through spatial style injection
• Progressive growing stabilizes training by gradually increasing resolution