Advanced Training
Data Quality and Curation â The Hidden Driver of LLM Performance
Data quality matters more than data quantity. A smaller, high-quality dataset consistently outperforms a larger, noisy one. This guide covers the science of data curation for LLMs.
- Deduplication â Removing duplicates can improve performance by 10%+ on downstream tasks
- Data Mixing â The ratio of code, math, books, and web text dramatically affects capabilities
- Quality Filtering â Perplexity-based and classifier-based filtering separate signal from noise
The quality of what you feed the model determines the quality of what it produces.
Data Quality and Curation for LLMs
Data quality is arguably the most important factor in LLM performance, yet it receives far less attention than model architecture or training algorithms. The "Scaling Data-Constrained Language Models" paper demonstrated that repeating data beyond 4 epochs yields diminishing returns, while carefully curated data can achieve the same performance with 10x fewer tokens.
The Data Quality Hierarchy
Level 1: Basic Filtering
Remove clearly low-quality data:
def basic_quality_filter(document):
if len(document) < 100 or len(document) > 100000:
return False
words = document.split()
if len(words) < 20:
return False
alpha_ratio = sum(c.isalpha() for c in document) / len(document)
if alpha_ratio < 0.8:
return False
word_repetition = len(words) / len(set(words))
if word_repetition > 3.0:
return False
return True
Level 2: Perplexity-Based Filtering
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
def compute_perplexity(doc, model, tokenizer, max_length=512):
inputs = tokenizer(doc, return_tensors="pt", truncation=True, max_length=max_length)
with torch.no_grad():
outputs = model(**inputs, labels=inputs["input_ids"])
return torch.exp(outputs.loss).item()
def perplexity_filter(documents, model_name="gpt2", low_thresh=10, high_thresh=1000):
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
filtered = []
for doc in documents:
ppl = compute_perplexity(doc, model, tokenizer)
if low_thresh <= ppl <= high_thresh:
filtered.append(doc)
return filtered
Level 3: Classifier-Based Filtering
def classifier_quality_score(document, classifier):
prob = classifier.predict_proba([document])[0][1]
return prob
high_quality_docs = [
doc for doc in corpus
if classifier_quality_score(doc, quality_classifier) > 0.7
]
Deduplication
Why Deduplication Matters
| Deduplication Method | Type | Precision | Speed |
|---|---|---|---|
| Exact hash | Exact | 100% | Very fast |
| MinHash LSH | Near-exact | ~95% | Fast |
| SimHash | Approximate | ~90% | Very fast |
| SemDeDup | Semantic | ~85% | Slow |
Exact Deduplication
import hashlib
def exact_dedup(documents):
seen_hashes = set()
unique_docs = []
for doc in documents:
doc_hash = hashlib.sha256(doc.encode()).hexdigest()
if doc_hash not in seen_hashes:
seen_hashes.add(doc_hash)
unique_docs.append(doc)
return unique_docs
Near-Duplicate Detection with MinHash
from datasketch import MinHash, MinHashLSH
def minhash_dedup(documents, threshold=0.8, num_perm=128):
lsh = MinHashLSH(threshold=threshold, num_perm=num_perm)
unique_docs = []
for i, doc in enumerate(documents):
m = MinHash(num_perm=num_perm)
for shingle in get_shingles(doc, k=5):
m.update(shingle.encode())
result = lsh.query(m)
if not result:
lsh.insert(str(i), m)
unique_docs.append(doc)
return unique_docs
Semantic Deduplication
from sentence_transformers import SentenceTransformer
from sklearn.cluster import DBSCAN
def semantic_dedup(documents, model_name="all-MiniLM-L6-v2", eps=0.3):
model = SentenceTransformer(model_name)
embeddings = model.encode(documents, show_progress_bar=True)
clustering = DBSCAN(eps=eps, min_samples=2, metric="cosine")
clusters = clustering.fit_predict(embeddings)
unique_docs = []
seen_clusters = set()
for doc, cluster_id in zip(documents, clusters):
if cluster_id == -1 or cluster_id not in seen_clusters:
unique_docs.append(doc)
if cluster_id != -1:
seen_clusters.add(cluster_id)
return unique_docs
Data Mixing
Domain Mixing Ratios
| Mixing Ratio (Web:Code:Books:Academic) | Coding | Reasoning | Knowledge | Instruction Following |
|---|---|---|---|---|
| 100:0:0:0 | Baseline | Baseline | Baseline | Baseline |
| 80:10:5:5 | +5% | +3% | +2% | +4% |
| 60:20:10:10 | +15% | +8% | +5% | +10% |
| 40:30:20:10 | +25% | +12% | +8% | +15% |
Optimal Mixing with DOREMI
def doremi_weights(domain_losses, target_loss):
weights = {}
for domain, loss in domain_losses.items():
weights[domain] = max(0, target_loss - loss)
total = sum(weights.values())
if total > 0:
weights = {k: v / total for k, v in weights.items()}
else:
weights = {k: 1.0 / len(weights) for k in weights}
return weights
Data Attribution
Measuring Data Influence
Data Quality Metrics
Comprehensive Quality Score
Practice Exercises
-
Deduplication Analysis: Given a corpus of 1M documents, estimate the size reduction after exact deduplication, MinHash LSH (threshold=0.8), and semantic deduplication. What is the typical deduplication rate for web crawl data?
-
Data Mixing Experiment: Design a data mixing strategy for an LLM that must excel at (a) conversational AI, (b) code generation, and (c) mathematical reasoning. Justify your domain proportions.
-
Quality Filter Design: Design a multi-stage quality filtering pipeline for Common Crawl data. What filters would you apply, in what order, and what retention rates would you target?
-
Data Attribution: How would you identify training examples that cause a specific hallucination in an LLM? Describe a practical approach using influence functions or similar methods.
Key Takeaways
What to Learn Next
-> Synthetic Data Generation Using LLMs to create high-quality training data for themselves.
-> Pretraining Language Models The fundamentals of training language models on large corpora.
-> Distributed Training for LLMs Scaling training across hundreds of GPUs with parallelism strategies.
-> Scaling Laws and Chinchilla How data quantity and quality interact with model scale.
-> Curriculum Learning for LLMs Strategic ordering of training data for improved learning.
-> Knowledge Distillation for LLMs Using teacher models to generate quality training signals.