Pix2Pix, CycleGAN, and Style Transfer

Module: Computer Vision | Difficulty: Premium

Pix2Pix Loss (Conditional GAN)

Cycle Consistency

Gram Matrix for Style

| Method | Paired | FID down | Diversity | Use Case | |--------|--------|----------|-----------|----------| | Pix2Pix | Yes | 80.7 | No | Paired translation | | CycleGAN | No | 72.3 | No | Unpaired translation | | MUNIT | No | 25.5 | Yes | Multi-modal | | CUT | No | 18.3 | No | Contrastive | | SPADE | Yes | 11.2 | Yes | Semantic synthesis |

import torch
import torch.nn as nn

class ResBlock(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.block = nn.Sequential(
            nn.Conv2d(channels, channels, 3, 1, 1, bias=False),
            nn.InstanceNorm2d(channels), nn.ReLU(inplace=True),
            nn.Conv2d(channels, channels, 3, 1, 1, bias=False),
            nn.InstanceNorm2d(channels),
        )
    def forward(self, x):
        return x + self.block(x)

class CycleGANGenerator(nn.Module):
    def __init__(self, channels=3, features=64, num_resnet=9):
        super().__init__()
        encoder = [
            nn.Conv2d(channels, features, 7, 1, 3, bias=False),
            nn.InstanceNorm2d(features), nn.ReLU(inplace=True),
            nn.Conv2d(features, features*2, 3, 2, 1, bias=False),
            nn.InstanceNorm2d(features*2), nn.ReLU(inplace=True),
            nn.Conv2d(features*2, features*4, 3, 2, 1, bias=False),
            nn.InstanceNorm2d(features*4), nn.ReLU(inplace=True),
        ]
        resnet = [ResBlock(features*4) for _ in range(num_resnet)]
        decoder = [
            nn.ConvTranspose2d(features*4, features*2, 3, 2, 1, 1, bias=False),
            nn.InstanceNorm2d(features*2), nn.ReLU(inplace=True),
            nn.ConvTranspose2d(features*2, features, 3, 2, 1, 1, bias=False),
            nn.InstanceNorm2d(features), nn.ReLU(inplace=True),
            nn.Conv2d(features, channels, 7, 1, 3),
            nn.Tanh(),
        ]
        self.encoder = nn.Sequential(*encoder)
        self.resnet = nn.Sequential(*resnet)
        self.decoder = nn.Sequential(*decoder)

    def forward(self, x):
        return self.decoder(self.resnet(self.encoder(x)))

Research Insight: Contrastive Unpaired Translation (CUT) eliminated the need for cycle consistency by using contrastive learning to align image patches between domains. This enables faster training and avoids the artifacts that cycle consistency can introduce when domains are not perfectly cycleable. Neural style transfer has evolved from slow optimization-based methods to feedforward networks, and now to diffusion-based methods that can disentangle content and style more effectively, enabling real-time high-quality stylization.