ControlNet: Spatial Conditioning

Module: Generative AI | Difficulty: Advanced

ControlNet Architecture

Zero Conolutions

Ensures ControlNet starts as identity, preserving pre-trained weights.

Training Strategy

1. Lock pre-trained U-Net
2. Train ControlNet with zero-init
3. Gradually increase scale from 0 to 1

Supported Conditions

class ZeroConv(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.conv = nn.Conv2d(channels, channels, 3, padding=1)
        nn.init.zeros_(self.conv.weight)
        nn.init.zeros_(self.conv.bias)
    def forward(self, x):
        return self.conv(x)

class ControlNet(nn.Module):
    def __init__(self, unet):
        super().__init__()
        self.zero_convs = nn.ModuleList([ZeroConv(c) for c in unet.channels])
        self.condition_encoder = nn.Sequential(
            nn.Conv2d(3, 64, 3, padding=1), nn.ReLU(),
            nn.Conv2d(64, 128, 3, 2, 1), nn.ReLU(),
            nn.Conv2d(128, 256, 3, 2, 1), nn.ReLU())

Research Insight: Zero convolutions are essential — without them, ControlNet can destabilize the pre-trained U-Net in the first few training steps, leading to mode collapse.