Alignment Tax and Capabilities

Alignment training shapes model behavior to be helpful, harmless, and honest. However, this shaping can reduce capabilities — the model may refuse to answer valid questions, provide less detailed responses, or perform worse on benchmarks.

Measuring Alignment Tax

Benchmark Comparison

def measure_alignment_tax(base_model, aligned_model, benchmarks):
    """Measure the capability difference between base and aligned models."""
    results = {}
    
    for benchmark in benchmarks:
        base_score = evaluate(base_model, benchmark)
        aligned_score = evaluate(aligned_model, benchmark)
        tax = (base_score - aligned_score) / base_score * 100
        
        results[benchmark.name] = {
            "base": base_score,
            "aligned": aligned_score,
            "tax_percent": tax
        }
    
    return results

Typical Alignment Tax

The Helpfulness-Harmlessness Tradeoff

def measure_tradeoff(model, helpful_prompts, harmful_prompts):
    """Measure the helpfulness-harmlessness tradeoff."""
    # Helpfulness: what percentage of legitimate questions are answered
    helpful_rate = 0
    for prompt in helpful_prompts:
        response = model.generate(prompt)
        if is_helpful(response):
            helpful_rate += 1
    helpful_rate /= len(helpful_prompts)
    
    # Harmlessness: what percentage of harmful requests are refused
    harmless_rate = 0
    for prompt in harmful_prompts:
        response = model.generate(prompt)
        if is_refused(response):
            harmless_rate += 1
    harmless_rate /= len(harmful_prompts)
    
    return {"helpfulness": helpful_rate, "harmlessness": harmless_rate}

Optimal Tradeoff Point

Preserving Capabilities

Selective Alignment

def selective_alignment(model, harmful_behaviors, preserve_capabilities):
    """Align model while preserving specific capabilities."""
    # Create targeted training data
    training_data = []
    
    # Add harmful behavior corrections
    for behavior in harmful_behaviors:
        training_data.extend(create_refusal_data(behavior))
    
    # Add capability preservation data
    for capability in preserve_capabilities:
        training_data.extend(create_capability_data(capability))
    
    # Train with lower learning rate to minimize capability drift
    aligned_model = dpo_train(model, training_data, lr=1e-6)
    
    return aligned_model

Capability Benchmarking

class CapabilityBenchmark:
    def __init__(self):
        self.benchmarks = {
            "reasoning": ReasoningBenchmark(),
            "coding": CodingBenchmark(),
            "math": MathBenchmark(),
            "knowledge": KnowledgeBenchmark(),
            "creativity": CreativityBenchmark(),
        }
    
    def full_evaluation(self, model):
        results = {}
        for name, benchmark in self.benchmarks.items():
            results[name] = benchmark.evaluate(model)
        return results

Alignment Tax Mitigation

Techniques for Reducing Tax

1. Multi-task training — Combine alignment with capability training
2. Curriculum learning — Align gradually, not all at once
3. Regularization — Penalize large deviations from base model
4. Selective alignment — Only align harmful behaviors
5. Data quality — High-quality alignment data reduces tax

def regularized_dpo(model, preference_data, reference_model, beta=0.1, lambda_reg=0.01):
    """DPO with regularization to minimize capability loss."""
    # Standard DPO loss
    dpo_loss = compute_dpo_loss(model, preference_data, reference_model, beta)
    
    # Regularization: penalize deviation from reference
    reg_loss = 0
    for param, ref_param in zip(model.parameters(), reference_model.parameters()):
        reg_loss += ((param - ref_param) ** 2).sum()
    
    total_loss = dpo_loss + lambda_reg * reg_loss
    return total_loss

Practice Exercises

1. Tax Measurement: Measure the alignment tax of a DPO-aligned model on 5 benchmarks. What is the average tax?

2. Tradeoff Analysis: Plot the helpfulness-harmlessness tradeoff curve by varying the beta parameter. What is the optimal tradeoff point?

3. Selective Alignment: Implement selective alignment that only targets harmful behaviors. Compare the tax to full alignment.

4. Regularization: Test different regularization strengths in DPO. How does regularization affect the alignment tax?

Key Takeaways

What to Learn Next