IP-Adapter: Image Prompt Conditioning

Module: Generative AI | Difficulty: Advanced

IP-Adapter Architecture

Decoupled Cross-Attention

Separate attention for text and image prompts:

Training

1. Train projection layer for CLIP image features
2. Fine-tune only cross-attention layers
3. Keep text encoder frozen

class IPAdapter(nn.Module):
    def __init__(self, unet, clip_vision, hidden_dim=768):
        super().__init__()
        self.clip = clip_vision
        self.image_proj = nn.Linear(clip_vision.output_dim, hidden_dim)
        self.to_k = nn.Linear(hidden_dim, unet.dim)
        self.to_v = nn.Linear(hidden_dim, unet.dim)
    def forward(self, x, t, text_cond, image_prompt):
        text_emb = text_cond
        image_emb = self.image_proj(self.clip(image_prompt))
        k = torch.cat([self.to_k(text_emb), self.to_k(image_emb)], dim=1)
        v = torch.cat([self.to_v(text_emb), self.to_v(image_emb)], dim=1)
        return self.unet(x, t, k, v)

Research Insight: IP-Adapter achieves 90% of DreamBooth quality with 0% fine-tuning. The key is that CLIP's image encoder already captures style and content information that can be directly injected via cross-attention.