Fine-Tuning LLMs with LoRA/QLoRA
PEFT + Transformers + Custom Datasets | Memory-Efficient Fine-Tuning
What You'll Learn
- 1Derive the trainable parameter count of a LoRA adapter for a given model, rank, and module set
- 2Configure 4-bit NF4 quantization and estimate the VRAM a QLoRA run needs before launching it
- 3Format instruction data correctly and split it without leaking near-duplicate rows into eval
- 4Evaluate a fine-tune with held-out loss, exact match, and perplexity before merging the adapter
Project Overview
Problem Statement
Fine-tuning a full 7B+ parameter LLM requires 28GB+ VRAM. LoRA and QLoRA enable fine-tuning on consumer GPUs by reducing trainable parameters by 90%+ while maintaining 95%+ performance.
Objectives
- Fine-tune LLaMA 3 8B on custom domain data using QLoRA
- Achieve measurable improvement on domain-specific benchmarks
- Implement efficient data preprocessing and formatting
- Deploy the fine-tuned model with proper evaluation
- Track experiments with MLflow and Weights & Biases
| Component | Technology |
|---|---|
| Base Model | LLaMA 3 8B / Mistral 7B |
| Fine-Tuning | PEFT (LoRA/QLoRA) |
| Training | Hugging Face Transformers + TRL |
| Quantization | bitsandbytes (4-bit NF4) |
| Dataset | Custom + Alpaca format |
| Evaluation | lm-evaluation-harness |
| Tracking | MLflow + W&B |
| Deployment | vLLM + Docker |
Design Rationale
LoRA works because the weight update a fine-tune learns is low rank. Instead of storing a full by update for every attention and MLP projection, the method freezes the base weights and learns two thin matrices whose product approximates the change:
The trainable count per projection collapses from to . On LLaMA 3 8B with for attention, an FFN width of 14,336, 32 layers, rank 16, and all seven projections targeted, the arithmetic is exact:
That is 45.1 million trainable parameters against 8.03 billion frozen ones, or 0.56 percent, which is why print_trainable_parameters reports a fraction of a percent even on an 8B model and why a single step costs a fraction of what full fine-tuning would.
QLoRA adds quantization so the frozen weights fit in VRAM. Storing 8.03 billion parameters in 4-bit NF4 costs about 4.1 GB including double quantization overhead, and AdamW keeps two fp32 moments plus fp32 gradients for the adapters only:
Roughly 541 MB of optimizer state, 4.1 GB of quantized weights, and a couple of gigabytes of activations land the whole run near 8 to 10 GB at sequence length 2,048 and micro-batch 4, so one 24 GB consumer card trains an 8B model with headroom for a larger batch.
Hyperparameters scale with the effective batch, which is micro-batch times gradient accumulation times GPU count:
A learning rate of 2e-4 tuned for an effective batch of 16 should rise roughly linearly if you double that batch, and the adapter scaling factor alpha divided by r, here 32 divided by 16, sets how strongly the low-rank update is applied relative to the frozen path.
Architecture Diagram
Step-by-Step Implementation
Step 1: Environment Setup
conda create -n llm-finetune python=3.10 -y
conda activate llm-finetune
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install transformers>=4.40.0 datasets accelerate
pip install peft>=0.10.0 trl>=0.8.0
pip install bitsandbytes>=0.43.0
pip install scipy sentencepiece protobuf
pip install mlflow wandb tensorboard
pip install lm-eval
wandb login
huggingface-cli login
Step 2: Data Preparation
The dataset preparation is critical for fine-tuning success. We support Alpaca, ShareGPT, and ChatML formats.
# src/data/prepare_dataset.py
import json
import hashlib
import logging
import pandas as pd
from dataclasses import dataclass
from typing import List, Dict, Optional
from datasets import Dataset, DatasetDict
from sklearn.model_selection import train_test_split
logger = logging.getLogger(__name__)
@dataclass
class DataConfig:
max_length: int = 2048
train_split: float = 0.9
eval_split: float = 0.1
seed: int = 42
min_response_length: int = 10
max_response_length: int = 2000
class DatasetPreparer:
def __init__(self, config: DataConfig):
self.config = config
def load_alpaca(self, path: str) -> List[Dict]:
with open(path) as f:
data = json.load(f)
formatted = []
for item in data:
text = self.format_alpaca(item)
if len(text) >= self.config.min_response_length:
formatted.append({"text": text})
return formatted
def format_alpaca(self, item: Dict) -> str:
if item.get("input"):
return (
f"Below is an instruction that describes a task.\n\n"
f"### Instruction:\n{item['instruction']}\n\n"
f"### Input:\n{item['input']}\n\n"
f"### Response:\n{item['output']}"
)
return (
f"Below is an instruction that describes a task.\n\n"
f"### Instruction:\n{item['instruction']}\n\n"
f"### Response:\n{item['output']}"
)
def create_dataset(self, data: List[Dict]) -> DatasetDict:
dataset = Dataset.from_list(data)
split = dataset.train_test_split(
test_size=self.config.eval_split,
seed=self.config.seed
)
return DatasetDict({
"train": split["train"],
"eval": split["test"],
})
Step 3: LoRA Configuration and Model Loading
Configure LoRA adapters targeting specific model layers for parameter-efficient fine-tuning.
# src/model/lora_config.py
from peft import LoraConfig, get_peft_model, TaskType, prepare_model_for_kbit_training
from transformers import (
AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig,
TrainingArguments
)
import torch
def load_model_with_lora(
model_name: str = "meta-llama/Meta-Llama-3-8B",
lora_r: int = 16,
lora_alpha: int = 32,
lora_dropout: float = 0.05,
use_4bit: bool = True,
):
# 4-bit quantization config
bnb_config = None
if use_4bit:
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
# Load base model
model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=bnb_config,
device_map="auto",
torch_dtype=torch.bfloat16,
trust_remote_code=True,
)
# Prepare for k-bit training
if use_4bit:
model = prepare_model_for_kbit_training(model)
# Configure LoRA
lora_config = LoraConfig(
task_type=TaskType.CAUSAL_LM,
r=lora_r,
lora_alpha=lora_alpha,
lora_dropout=lora_dropout,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
bias="none",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"
return model, tokenizer
Step 4: Training with SFTTrainer
Use the TRL SFTTrainer for efficient supervised fine-tuning with PEFT integration.
# src/train.py
from transformers import TrainingArguments, TrainerCallback
from trl import SFTTrainer, SFTConfig
from peft import PeftModel
import mlflow
import os
class MLflowCallback(TrainerCallback):
def on_log(self, args, state, control, logs=None, **kwargs):
if logs:
for k, v in logs.items():
if isinstance(v, (int, float)):
mlflow.log_metric(k, v, step=state.global_step)
def on_save(self, args, state, control, **kwargs):
mlflow.log_metric("checkpoint_step", state.global_step)
def train(model, tokenizer, dataset, output_dir="./output"):
training_args = SFTConfig(
output_dir=output_dir,
num_train_epochs=3,
per_device_train_batch_size=4,
per_device_eval_batch_size=4,
gradient_accumulation_steps=4,
learning_rate=2e-4,
weight_decay=0.01,
warmup_ratio=0.03,
lr_scheduler_type="cosine",
logging_steps=10,
eval_strategy="steps",
eval_steps=100,
save_steps=200,
save_total_limit=3,
bf16=True,
max_seq_length=2048,
dataset_text_field="text",
report_to="none",
optim="paged_adamw_32bit",
max_grad_norm=0.3,
group_by_length=True,
)
mlflow.set_experiment("llm-finetuning")
with mlflow.start_run(run_name="qlora-llama3-8b"):
mlflow.log_params({
"model": "llama-3-8b",
"lora_r": 16,
"lora_alpha": 32,
"learning_rate": 2e-4,
"epochs": 3,
"quantization": "qlora-nf4",
})
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=dataset["train"],
eval_dataset=dataset["eval"],
processing_class=tokenizer,
callbacks=[MLflowCallback()],
)
trainer.train()
# Save LoRA adapter
trainer.save_model(os.path.join(output_dir, "lora-adapter"))
mlflow.log_artifact(os.path.join(output_dir, "lora-adapter"))
return trainer
Step 5: Model Evaluation
Evaluate the fine-tuned model on held-out test data using multiple metrics.
# src/evaluate.py
import torch
import numpy as np
from sklearn.metrics import accuracy_score, f1_score
from transformers import pipeline
def evaluate_model(model, tokenizer, test_data, metrics=None):
pipe = pipeline(
"text-generation",
model=model,
tokenizer=tokenizer,
max_new_tokens=256,
temperature=0.1,
)
predictions = []
references = []
for item in test_data:
prompt = f"### Instruction:\n{item['instruction']}\n\n### Response:\n"
output = pipe(prompt)[0]["generated_text"]
pred = output.split("### Response:\n")[-1].strip()
predictions.append(pred)
references.append(item["output"])
results = {}
results["exact_match"] = accuracy_score(references, predictions)
print(f"Evaluation Results:")
print(f" Exact Match: {results['exact_match']:.4f}")
return results
def compute_perplexity(model, tokenizer, texts):
encodings = tokenizer(texts, return_tensors="pt", padding=True, truncation=True)
with torch.no_grad():
outputs = model(**encodings, labels=encodings["input_ids"])
return torch.exp(outputs.loss).item()
Step 6: Merge and Export
Merge LoRA adapters back into the base model for optimized inference.
# src/merge_export.py
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
def merge_and_export(
base_model_name: str,
adapter_path: str,
output_path: str,
):
base_model = AutoModelForCausalLM.from_pretrained(
base_model_name,
torch_dtype=torch.bfloat16,
device_map="auto",
)
model = PeftModel.from_pretrained(base_model, adapter_path)
model = model.merge_and_unload()
model.save_pretrained(output_path)
tokenizer = AutoTokenizer.from_pretrained(base_model_name)
tokenizer.save_pretrained(output_path)
print(f"Merged model saved to {output_path}")
Step 7: Deployment with vLLM
Deploy the merged model using vLLM for high-throughput inference.
# Dockerfile
FROM vllm/vllm-openai:latest
COPY merged_model/ /models/llama3-8b-qlora
CMD ["python", "-m", "vllm.entrypoints.openai.api_server",
"--model", "/models/llama3-8b-qlora",
"--tensor-parallel-size", "1",
"--max-model-len", "4096",
"--gpu-memory-utilization", "0.9"]
# Deploy with Docker
docker build -t llama3-qlora-served .
docker run -p 8000:8000 --gpus all llama3-qlora-served
Test the Inference Endpoint
curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{"model": "llama3-8b-qlora", "prompt": "Hello!", "max_tokens": 100}'
âšī¸
Always save checkpoints during training. Fine-tuning runs can be expensive and time-consuming. Use MLflow or W&B to track all experiments and reproduce results.
đĄ
Start with a smaller model and dataset for rapid prototyping. Only scale up to larger models once your pipeline is validated.
Performance Metrics
| Metric | Target | Description |
|---|---|---|
| Training Loss | Converging | Should decrease steadily |
| Validation Loss | Minimal gap | No overfitting |
| Domain Accuracy | > 90% | On benchmark tasks |
| Inference Latency | < 100ms | Per token generation |
| Memory Usage | < 16GB | Fits single GPU |
What Breaks in Production
Fine-tuning fails quietly far more often than it crashes. The first failure is format mismatch: a model trained on Alpaca blocks but served without the Alpaca markers degrades sharply, because those markers are part of what the adapter learned. Persist the chat template alongside the adapter and replay it verbatim at inference.
The second failure is leakage in evaluation. Randomly splitting 10,000 instruction rows puts near duplicates of the same source article in both train and eval and inflates exact match by 5 to 10 points, so split by source document or by timestamp instead of by row.
The third is forgetting from too many epochs. Three passes over 10,000 curated examples is usually safe; ten passes drives train loss down while eval loss turns upward and general capability collapses. Watch the gap directly:
Once the gap grows past 0.3 while train loss is still falling, stop and regularize with dropout or more data rather than training longer.
Serving has its own trap: merging the adapter writes a new checkpoint, and a vLLM server pointed at the base model silently ignores your work. Point it at the merged directory, pin the model revision, and smoke-test with a domain prompt the base model is known to get wrong.
Interview Talking Points
- Parameter Efficiency: LoRA reduces trainable parameters from billions to millions by learning low-rank decomposition matrices.
- QLoRA Innovation: Quantizing the base model to 4-bit NF4 while computing gradients in 16-bit BFloat16 enables fine-tuning 65B models on a single 48GB GPU.
- Data Quality: High-quality instruction-following data is more important than quantity. 10K curated examples often outperform 100K noisy ones.
- Hyperparameter Sensitivity: Learning rate (1e-4 to 3e-4), rank (8-64), and target modules significantly impact results.
- Evaluation Strategy: Combine automatic metrics with human evaluation and LLM-as-judge approaches.
- Deployment: Merge LoRA adapters into the base model for faster inference. Use vLLM or TGI for serving.
â ī¸
Always save checkpoints during training. Fine-tuning runs can be expensive and time-consuming. Use MLflow or W&B to track all experiments.
âšī¸
This project demonstrates production-grade LLM fine-tuning. For the complete implementation, refer to the accompanying repository.
Common Mistakes
Training on one prompt format and serving with another
Fix: Save the chat template with the adapter and reuse it byte for byte at inference.
Copying a rank value from a tutorial without budgeting it
Fix: Compute r times (d_in + d_out) per target module and raise rank only when eval loss plateaus.
Merging the adapter, then loading the base model for serving
Fix: Serve the merged directory or load the adapter explicitly, then verify with a known-hard prompt.
Best Practices
- Log trainable parameter percentage and peak VRAM at step zero so config drift shows up immediately
- Split eval data by source document or timestamp instead of random rows
- Checkpoint every 200 steps and keep the three best runs by eval loss
Key Takeaways
- LoRA learns W = W0 + BA, cutting trainable parameters to r times (d_in + d_out) per matrix
- At rank 16 across seven projections in 32 layers, LLaMA 3 8B trains 45.1M parameters, 0.56 percent
- 4-bit NF4 puts an 8B base model near 4.1 GB, so a 24 GB card fits weights plus optimizer state
- Effective batch is micro-batch times accumulation times GPUs, and learning rate must track it
Frequently Asked Questions
What rank should I start with?
Rank 16 is the pragmatic default: it captures most of the gain on 7B to 8B models while keeping the adapter under 50 MB. Drop to 8 when VRAM is tight, and try 32 or 64 only after you have evidence that eval loss is plateaued for a rank-16 run on the same data.
Does QLoRA lose quality against full precision fine-tuning?
Benchmarks consistently show 4-bit NF4 within about 1 percent of 16-bit full fine-tuning on downstream tasks, because the quantization applies only to the frozen base while adapters and gradients train in higher precision. The trade is a slower step time from dequantization, not accuracy.
When should you merge the adapter instead of loading it separately?
Merge when you serve a single model and want maximum throughput, because one checkpoint avoids the extra memory and kernel dispatch of a live adapter. Keep them separate when you need many adapters on one base, A/B testing, or fast rollback to the unmodified model.
Interview Questions
Q1: Why can QLoRA fine-tune a 65B model on a single 48 GB GPU?
The base weights are quantized to 4-bit NF4, so 65 billion parameters occupy roughly 33 GB, and only the low-rank adapters carry gradients and optimizer state. At rank 64 that is a few hundred million parameters, leaving room for activations.
Q2: How do you detect catastrophic forgetting during a run?
Track the train and eval loss gap plus a small general benchmark evaluated every epoch. When eval loss rises while train loss falls, or the general benchmark drops more than a point or two, the adapter is overwriting capabilities it should keep.
Q3: Why does the instruction format matter as much as the data itself?
The model learns the markers as control tokens, so ### Instruction and ### Response become part of the policy. Serving without them shifts the input distribution away from training and can cost 5 to 10 points of accuracy on identical content.