LoRA and PEFT

Parameter-Efficient Fine-Tuning (PEFT) methods enable adaptation of large language models by updating only a small subset of parameters. This tutorial provides a rigorous treatment of LoRA, its variants, and practical implementation.

LoRA: Low-Rank Adaptation

LoRA (Hu et al., 2021) is the most widely used PEFT method. It freezes the pre-trained weights and injects trainable low-rank decomposition matrices into each layer of the Transformer.

LoRA Decomposition

The forward pass with LoRA becomes:

Parameter Efficiency

For a 7B model with r=16, applying LoRA to Q and V projections:

• Full parameters: 7B
• LoRA parameters: 2 x 16 x 4096 x 32 x 2 = 8.4M (0.12% of total)

Which Layers to Adapt?

Common strategies for applying LoRA:

• Q, V projections (default): Most impactful, fewest parameters
• Q, K, V, O projections: Full attention adaptation
• Q, K, V, O + FFN: Maximum expressiveness
• All linear layers: Maximum coverage

LoRA Initialization

The initialization strategy is crucial for LoRA performance:

import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, original_linear, r=8, alpha=16):
        super().__init__()
        self.original = original_linear
        self.original.weight.requires_grad = False
        self.original.bias.requires_grad = False
        
        d_out, d_in = original_linear.weight.shape
        self.lora_A = nn.Parameter(torch.randn(r, d_in) * (1 / d_in ** 0.5))
        self.lora_B = nn.Parameter(torch.zeros(d_out, r))
        self.scaling = alpha / r
        
    def forward(self, x):
        base_out = self.original(x)
        lora_out = (x @ self.lora_A.T @ self.lora_B.T) * self.scaling
        return base_out + lora_out

QLoRA: Quantized LoRA

QLoRA (Dettmers et al., 2023) combines 4-bit quantization with LoRA, enabling fine-tuning of large models on consumer GPUs.

Memory Savings

NF4 Quantization

LoRA Variants

AdaLoRA

AdaLoRA (Zhang et al., 2023) adapts the rank allocation across layers based on importance:

DoRA

DoRA (Liu et al., 2024) decomposes weights into magnitude and direction:

HuggingFace PEFT Library

from peft import (
    LoraConfig,
    get_peft_model,
    TaskType,
    PeftModel
)
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "meta-llama/Llama-2-7b-hf"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    lora_dropout=0.1,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    bias="none",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 13,631,488 || all params: 6,742,609,920 || trainable%: 0.2022

# Train the model...
# trainer.train()

# Save only LoRA weights
model.save_pretrained("./lora-adapter")

# Load and merge later
base_model = AutoModelForCausalLM.from_pretrained(model_name)
model = PeftModel.from_pretrained(base_model, "./lora-adapter")
merged_model = model.merge_and_unload()

When to Use LoRA vs Full Fine-tuning

Practice Exercises

1. Mathematical: Calculate the number of trainable parameters for LoRA applied to Q and V projections of a model with 32 layers, d_model=4096, and rank r=16. Compare with the total model parameters.

2. Implementation: Use the HuggingFace PEFT library to apply LoRA to Mistral-7B and fine-tune on a small dataset. Compare training time and memory with full fine-tuning.

3. Analysis: Experiment with different LoRA ranks (4, 8, 16, 32, 64). Plot the relationship between rank, trainable parameters, and task performance.

4. Research: Compare LoRA, QLoRA, and DoRA on the same task. Which provides the best trade-off between performance and efficiency?

What to Learn Next