Data Quality and Curation for LLMs

Data quality is arguably the most important factor in LLM performance, yet it receives far less attention than model architecture or training algorithms. The "Scaling Data-Constrained Language Models" paper demonstrated that repeating data beyond 4 epochs yields diminishing returns, while carefully curated data can achieve the same performance with 10x fewer tokens.

The Data Quality Hierarchy

Level 1: Basic Filtering

Remove clearly low-quality data:

def basic_quality_filter(document):
    if len(document) < 100 or len(document) > 100000:
        return False
    words = document.split()
    if len(words) < 20:
        return False
    alpha_ratio = sum(c.isalpha() for c in document) / len(document)
    if alpha_ratio < 0.8:
        return False
    word_repetition = len(words) / len(set(words))
    if word_repetition > 3.0:
        return False
    return True

Level 2: Perplexity-Based Filtering

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

def compute_perplexity(doc, model, tokenizer, max_length=512):
    inputs = tokenizer(doc, return_tensors="pt", truncation=True, max_length=max_length)
    with torch.no_grad():
        outputs = model(**inputs, labels=inputs["input_ids"])
    return torch.exp(outputs.loss).item()

def perplexity_filter(documents, model_name="gpt2", low_thresh=10, high_thresh=1000):
    model = AutoModelForCausalLM.from_pretrained(model_name)
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    filtered = []
    for doc in documents:
        ppl = compute_perplexity(doc, model, tokenizer)
        if low_thresh <= ppl <= high_thresh:
            filtered.append(doc)
    return filtered

Level 3: Classifier-Based Filtering

def classifier_quality_score(document, classifier):
    prob = classifier.predict_proba([document])[0][1]
    return prob

high_quality_docs = [
    doc for doc in corpus
    if classifier_quality_score(doc, quality_classifier) > 0.7
]

Deduplication

Why Deduplication Matters

Exact Deduplication

import hashlib

def exact_dedup(documents):
    seen_hashes = set()
    unique_docs = []
    for doc in documents:
        doc_hash = hashlib.sha256(doc.encode()).hexdigest()
        if doc_hash not in seen_hashes:
            seen_hashes.add(doc_hash)
            unique_docs.append(doc)
    return unique_docs

Near-Duplicate Detection with MinHash

from datasketch import MinHash, MinHashLSH

def minhash_dedup(documents, threshold=0.8, num_perm=128):
    lsh = MinHashLSH(threshold=threshold, num_perm=num_perm)
    unique_docs = []
    for i, doc in enumerate(documents):
        m = MinHash(num_perm=num_perm)
        for shingle in get_shingles(doc, k=5):
            m.update(shingle.encode())
        result = lsh.query(m)
        if not result:
            lsh.insert(str(i), m)
            unique_docs.append(doc)
    return unique_docs

Semantic Deduplication

from sentence_transformers import SentenceTransformer
from sklearn.cluster import DBSCAN

def semantic_dedup(documents, model_name="all-MiniLM-L6-v2", eps=0.3):
    model = SentenceTransformer(model_name)
    embeddings = model.encode(documents, show_progress_bar=True)
    clustering = DBSCAN(eps=eps, min_samples=2, metric="cosine")
    clusters = clustering.fit_predict(embeddings)
    unique_docs = []
    seen_clusters = set()
    for doc, cluster_id in zip(documents, clusters):
        if cluster_id == -1 or cluster_id not in seen_clusters:
            unique_docs.append(doc)
            if cluster_id != -1:
                seen_clusters.add(cluster_id)
    return unique_docs

Data Mixing

Domain Mixing Ratios

Optimal Mixing with DOREMI

def doremi_weights(domain_losses, target_loss):
    weights = {}
    for domain, loss in domain_losses.items():
        weights[domain] = max(0, target_loss - loss)
    total = sum(weights.values())
    if total > 0:
        weights = {k: v / total for k, v in weights.items()}
    else:
        weights = {k: 1.0 / len(weights) for k in weights}
    return weights

Data Attribution

Measuring Data Influence

Data Quality Metrics

Comprehensive Quality Score

Practice Exercises

1. Deduplication Analysis: Given a corpus of 1M documents, estimate the size reduction after exact deduplication, MinHash LSH (threshold=0.8), and semantic deduplication. What is the typical deduplication rate for web crawl data?

2. Data Mixing Experiment: Design a data mixing strategy for an LLM that must excel at (a) conversational AI, (b) code generation, and (c) mathematical reasoning. Justify your domain proportions.

3. Quality Filter Design: Design a multi-stage quality filtering pipeline for Common Crawl data. What filters would you apply, in what order, and what retention rates would you target?

4. Data Attribution: How would you identify training examples that cause a specific hallucination in an LLM? Describe a practical approach using influence functions or similar methods.

Key Takeaways

What to Learn Next