Personalization: DreamBooth, LoRA, and Customization

Module: Generative AI | Difficulty: Advanced

DreamBooth

Fine-tune entire model on 3-5 images:

LoRA (Low-Rank Adaptation)

where .

Textual Inversion

Learn a new token in the CLIP text encoder:

import torch, torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, in_f, out_f, rank=4, alpha=1.0):
        super().__init__()
        self.linear = nn.Linear(in_f, out_f, bias=False)
        self.linear.weight.requires_grad = False
        self.lora_A = nn.Parameter(torch.randn(in_f, rank) * 0.01)
        self.lora_B = nn.Parameter(torch.zeros(rank, out_f))
        self.scaling = alpha / rank
    def forward(self, x):
        return self.linear(x) + (x @ self.lora_A @ self.lora_B) * self.scaling

| Method | Params | Training Time | Quality | |--------|--------|---------------|---------| | DreamBooth | 860M | 5 min | Excellent | | LoRA | 2M | 3 min | Very Good | | Textual Inversion | 1 | 10 min | Good | | Custom Diffusion | 4M | 4 min | Very Good |

Research Insight: LoRA works as well as full fine-tuning for personalization because the task-specific information has low intrinsic dimensionality. The optimal rank is typically 4-8.