Reference-Only Generation

Module: Generative AI | Difficulty: Advanced

Self-Attention Injection

Style Mixing

Inject different layers for different aspects:

• Early layers: structure
• Middle layers: style
• Late layers: texture

Reference-Only ControlNet

def reference_attention(gen_features, ref_features, layer_idx):
    q = gen_features.mean(dim=1)  # [B, C]
    k = ref_features.mean(dim=1)  # [B, C]
    v = ref_features
    attn = torch.softmax(q @ k.T / q.size(-1)**0.5, dim=-1)
    return (attn @ v).mean(dim=1)

Research Insight: Reference-only generation is more effective than IP-Adapter for style transfer because it uses the full image features rather than CLIP's compressed representation. However, it requires more VRAM.