Controllable Generation: ControlNet and Beyond

Module: Generative AI | Difficulty: Advanced

ControlNet

Add conditioning to frozen U-Net:

Zero Conolutions

Initialize with zeros for training stability.

T2I-Adapter

Lightweight adapter (~77M params) vs ControlNet (~361M params).

IP-Adapter

Decoupled cross-attention:

class ZeroConv(nn.Module):
    def __init__(self, ch):
        super().__init__()
        self.conv = nn.Conv2d(ch, ch, 3, padding=1)
        nn.init.zeros_(self.conv.weight)
        nn.init.zeros_(self.conv.bias)
    def forward(self, x): return self.conv(x)

| Method | Params | Control Types | Speed | |--------|--------|---------------|-------| | ControlNet | 361M | Edge, depth, pose | 1.0x | | T2I-Adapter | 77M | Edge, color, depth | 1.2x | | IP-Adapter | 220M | Image reference | 1.1x |