Fine-Tuning Methods: LoRA, Adapters, and Prompt Tuning
Module: Natural Language Processing | Difficulty: Advanced
Full Fine-Tuning
LoRA
Adapters
Prompt Tuning
Comparison
| Method | Trainable Params | Memory | Performance |
|---|---|---|---|
| Full Fine-Tune | 100% | High | Baseline |
| LoRA | 0.1-1% | Low | 95-99% |
| Adapter | 1-5% | Medium | 93-97% |
| Prompt Tuning | 0.01% | Very Low | 85-95% |
import torch
import torch.nn as nn
class LoRALinear(nn.Module):
def __init__(self, in_features, out_features, rank=8, alpha=1.0):
super().__init__()
self.linear = nn.Linear(in_features, out_features, bias=False)
self.linear.weight.requires_grad = False
self.lora_A = nn.Parameter(torch.randn(in_features, rank) * 0.01)
self.lora_B = nn.Parameter(torch.zeros(rank, out_features))
self.scaling = alpha / rank
def forward(self, x):
return self.linear(x) + (x @ self.lora_A @ self.lora_B) * self.scaling
class PromptTuning(nn.Module):
def __init__(self, n_prompts=10, d_model=768):
super().__init__()
self.prompt_embeddings = nn.Parameter(torch.randn(n_prompts, d_model) * 0.01)
def forward(self, batch_size):
return self.prompt_embeddings.unsqueeze(0).expand(batch_size, -1, -1)
Research Insight: LoRA's effectiveness comes from the low-rank structure of weight updates. Task-specific information has low intrinsic dimensionality, so rank 4-8 captures most of the task-relevant information while using 100x fewer parameters.