Image Editing with Diffusion Models

Module: Generative AI | Difficulty: Advanced

InstructPix2Pix

Plug-and-Play (PnP)

Inject self-attention from reference image:

Attention Injection

def attention_injection(q_edit, k_ref, v_ref, scale=1.0):
    attn = torch.softmax(q_edit @ k_ref.T / (q_edit.size(-1)**0.5), dim=-1)
    return attn @ v_ref * scale

Research Insight: Attention injection preserves the structure of the original image while changing the style/content. The key is injecting at the right layers — too early loses structure, too late loses editability.