LLM Fine-Tuning: LoRA, QLoRA, and Full Fine-Tuning Compared
Fine-tuning large language models has become more accessible with techniques like LoRA and QLoRA, but choosing the right approach depends on your budget, dataset size, and performance requirements. This guide compares all major fine-tuning approaches with practical implementation code.
The Fine-Tuning Spectrum
Sponsored
Architecture Diagram
Full Fine-Tuning ←→ Parameter-Efficient Fine-Tuning (PEFT)
↓ ↓
Most expensive, Cheapest, fastest
best quality good quality
Full Fine-Tuning
Updates all model parameters. Requires the most resources but achieves the best results.
python
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
TrainingArguments,
Trainer
)
# Load base model
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-8B",
torch_dtype=torch.bfloat16,
device_map="auto"
)
# Training configuration
training_args = TrainingArguments(
output_dir="./output",
num_train_epochs=3,
per_device_train_batch_size=4,
gradient_accumulation_steps=8,
learning_rate=2e-5,
weight_decay=0.01,
warmup_ratio=0.03,
lr_scheduler_type="cosine",
fp16=True,
logging_steps=10,
save_strategy="epoch",
)
# Train
trainer = Trainer(
model=model,
args=training_args,
train_dataset=dataset,
tokenizer=tokenizer,
)
trainer.train()
Requirements
- GPU: 4x A100 80GB (for 8B model)
- RAM: 128GB+
- Storage: 500GB+ (model checkpoints)
- Cost: ~$50-100/hour
- Time: 4-24 hours depending on dataset
LoRA (Low-Rank Adaptation)
Freezes the original model and trains small rank-decomposition matrices. Reduces trainable parameters by 90%+.
python
from peft import LoraConfig, get_peft_model, TaskType
# Configure LoRA
lora_config = LoraConfig(
task_type=TaskType.CAUSAL_LM,
r=16, # Rank (higher = more capacity)
lora_alpha=32, # Scaling factor
lora_dropout=0.1, # Dropout for regularization
target_modules=[ # Which layers to adapt
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"
],
bias="none",
)
# Apply LoRA to model
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-8B",
torch_dtype=torch.bfloat16,
device_map="auto"
)
model = get_peft_model(model, lora_config)
# Print trainable parameters
model.print_trainable_parameters()
# Output: trainable params: 41,943,040 || all params: 8,071,319,552 || 0.52%
# Train (same as before, but much faster and cheaper)
trainer = Trainer(model=model, args=training_args, ...)
trainer.train()
# Save only the LoRA weights (~100MB instead of 16GB)
model.save_pretrained("./lora_weights")
# Load for inference
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B")
model = PeftModel.from_pretrained(base_model, "./lora_weights")
LoRA Hyperparameters
| Parameter | Range | Effect |
|---|---|---|
r (rank) | 4-64 | Higher = more capacity, more parameters |
lora_alpha | 8-64 | Scaling factor (usually 2x rank) |
lora_dropout | 0.05-0.2 | Regularization |
target_modules | varies | Which layers to adapt |
QLoRA (Quantized LoRA)
Sponsored
LoRA with 4-bit quantization. Fine-tune 70B models on a single 48GB GPU.
python
from peft import LoraConfig, get_peft_model
from transformers import BitsAndBytesConfig
# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4", # NormalFloat4 quantization
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True, # Nested quantization
)
# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-70B",
quantization_config=bnb_config,
device_map="auto"
)
# Apply LoRA (same as before)
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
lora_dropout=0.05,
bias="none",
task_type=TaskType.CAUSAL_LM,
)
model = get_peft_model(model, lora_config)
# Train with gradient checkpointing (saves memory)
training_args = TrainingArguments(
per_device_train_batch_size=2,
gradient_accumulation_steps=16,
gradient_checkpointing=True, # Critical for QLoRA
...
)
QLoRA vs LoRA Performance
| Metric | LoRA (16-bit) | QLoRA (4-bit) | Difference |
|---|---|---|---|
| Model quality | Baseline | -0.5-1% | Negligible |
| Memory usage | 32GB | 12GB | 62% less |
| Training speed | 1x | 0.7x | 30% slower |
| Inference speed | 1x | 0.9x | 10% slower |
RLHF (Reinforcement Learning from Human Feedback)
Fine-tune models to align with human preferences.
python
from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead
# Load model with value head for PPO
model = AutoModelForCausalLMWithValueHead.from_pretrained(
"meta-llama/Llama-3.1-8B",
torch_dtype=torch.bfloat16
)
# PPO configuration
ppo_config = PPOConfig(
learning_rate=1.4e-5,
batch_size=64,
mini_batch_size=16,
ppo_epochs=4,
kl_penalty="kl", # KL divergence penalty
init_kl_coef=0.2,
target_kl=6.0,
gamma=1.0,
lam=0.95,
cliprange=0.2,
)
# Reward model
reward_model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-8B-reward"
)
# RLHF training loop
ppo_trainer = PPOTrainer(
config=ppo_config,
model=model,
ref_model=None, # Uses model's own copy
tokenizer=tokenizer,
)
for batch in dataloader:
query_tensors = [tokenizer.encode(q) for q in batch['queries']]
# Generate responses
response_tensors = ppo_trainer.generate(query_tensors)
# Get rewards
rewards = [reward_model(r) for r in response_tensors]
# PPO update
stats = ppo_trainer.step(query_tensors, response_tensors, rewards)
DPO (Direct Preference Optimization)
Simpler alternative to RLHF — no reward model needed.
python
from trl import DPOTrainer, DPOConfig
# DPO training
dpo_config = DPOConfig(
beta=0.1, # KL penalty weight
loss_type="sigmoid", # DPO loss variant
learning_rate=5e-7,
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
)
trainer = DPOTrainer(
model=model,
ref_model=ref_model,
args=dpo_config,
train_dataset=preference_dataset, # (chosen, rejected) pairs
tokenizer=tokenizer,
)
trainer.train()
RLHF vs DPO Comparison
| Aspect | RLHF | DPO |
|---|---|---|
| Complexity | High (3 models) | Low (2 models) |
| Training stability | Can be unstable | More stable |
| Reward hacking | Possible | Less likely |
| Performance | Slightly better | Very competitive |
| Compute cost | Higher | Lower |
Cost Comparison
| Method | GPU Required | Time (10K examples) | Cost |
|---|---|---|---|
| Full Fine-Tuning | 4x A100 80GB | 4-8 hours | $200-400 |
| LoRA (8B) | 1x A100 80GB | 2-4 hours | $50-100 |
| QLoRA (8B) | 1x A100 40GB | 3-6 hours | $30-60 |
| QLoRA (70B) | 1x A100 80GB | 8-16 hours | $100-200 |
| DPO | 2x A100 80GB | 4-8 hours | $100-200 |
| RLHF | 4x A100 80GB | 12-24 hours | $300-600 |
When to Use Each Method
Advertisement
Use Full Fine-Tuning when:
- You have a large dataset (100K+ examples)
- Maximum quality is critical
- Budget allows for significant compute
- You're fine-tuning a small model (<3B parameters)
Use LoRA when:
- You have a medium dataset (10K-100K examples)
- You need good quality with lower cost
- You want to preserve base model capabilities
- You need to fine-tune for multiple tasks
Use QLoRA when:
- You have limited GPU budget
- You're fine-tuning a large model (8B-70B)
- You want to experiment quickly
- Training speed is less critical
Use DPO/RLHF when:
- You want to align model with human preferences
- You have preference data (chosen/rejected pairs)
- Safety and alignment are important
Production Deployment
Serving LoRA Models
python
# Option 1: Merge LoRA weights into base model
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B")
model = PeftModel.from_pretrained(base_model, "./lora_weights")
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./merged_model")
# Option 2: Serve with dynamic LoRA switching
class LoRAServing:
def __init__(self, base_model):
self.base_model = base_model
self.lora_weights = {} # name -> weights
def load_lora(self, name: str, path: str):
self.lora_weights[name] = PeftModel.from_pretrained(
self.base_model, path
)
def generate(self, prompt: str, adapter: str = None):
if adapter:
model = self.lora_weights[adapter]
else:
model = self.base_model
return model.generate(prompt)
Conclusion
The choice of fine-tuning method depends on your specific constraints:
- QLoRA is the best starting point for most teams — low cost, good quality
- LoRA is ideal for production systems needing multiple adapters
- Full fine-tuning is for maximum quality when budget allows
- DPO is the simplest way to align with human preferences
- RLHF is for cutting-edge alignment when DPO isn't enough
The trend is toward parameter-efficient methods. With LoRA and QLoRA, you can fine-tune state-of-the-art models on a single GPU.