Alignment
Alignment Tax and Capabilities â The Cost of Safety
Alignment training improves safety but can reduce model capabilities. Understanding this tradeoff â the "alignment tax" â is crucial for building models that are both safe and useful.
- Capability Degradation â How alignment affects benchmark performance
- Helpfulness vs Harmlessness â The fundamental alignment tension
- Measuring Tax â Quantifying what alignment costs
Safety without capability is useless; capability without safety is dangerous.
Alignment Tax and Capabilities
Alignment training shapes model behavior to be helpful, harmless, and honest. However, this shaping can reduce capabilities â the model may refuse to answer valid questions, provide less detailed responses, or perform worse on benchmarks.
Measuring Alignment Tax
Benchmark Comparison
def measure_alignment_tax(base_model, aligned_model, benchmarks):
"""Measure the capability difference between base and aligned models."""
results = {}
for benchmark in benchmarks:
base_score = evaluate(base_model, benchmark)
aligned_score = evaluate(aligned_model, benchmark)
tax = (base_score - aligned_score) / base_score * 100
results[benchmark.name] = {
"base": base_score,
"aligned": aligned_score,
"tax_percent": tax
}
return results
Typical Alignment Tax
| Benchmark | Base Model | Aligned Model | Tax (%) |
|---|---|---|---|
| MMLU | 78.5 | 76.2 | 2.9% |
| HumanEval | 67.0 | 62.5 | 6.7% |
| GSM8K | 85.0 | 82.3 | 3.2% |
| TruthfulQA | 45.0 | 62.0 | -37.8% (improvement!) |
| BBQ (bias) | 35.0 | 55.0 | -57.1% (improvement!) |
The Helpfulness-Harmlessness Tradeoff
def measure_tradeoff(model, helpful_prompts, harmful_prompts):
"""Measure the helpfulness-harmlessness tradeoff."""
# Helpfulness: what percentage of legitimate questions are answered
helpful_rate = 0
for prompt in helpful_prompts:
response = model.generate(prompt)
if is_helpful(response):
helpful_rate += 1
helpful_rate /= len(helpful_prompts)
# Harmlessness: what percentage of harmful requests are refused
harmless_rate = 0
for prompt in harmful_prompts:
response = model.generate(prompt)
if is_refused(response):
harmless_rate += 1
harmless_rate /= len(harmful_prompts)
return {"helpfulness": helpful_rate, "harmlessness": harmless_rate}
Optimal Tradeoff Point
Preserving Capabilities
Selective Alignment
def selective_alignment(model, harmful_behaviors, preserve_capabilities):
"""Align model while preserving specific capabilities."""
# Create targeted training data
training_data = []
# Add harmful behavior corrections
for behavior in harmful_behaviors:
training_data.extend(create_refusal_data(behavior))
# Add capability preservation data
for capability in preserve_capabilities:
training_data.extend(create_capability_data(capability))
# Train with lower learning rate to minimize capability drift
aligned_model = dpo_train(model, training_data, lr=1e-6)
return aligned_model
Capability Benchmarking
class CapabilityBenchmark:
def __init__(self):
self.benchmarks = {
"reasoning": ReasoningBenchmark(),
"coding": CodingBenchmark(),
"math": MathBenchmark(),
"knowledge": KnowledgeBenchmark(),
"creativity": CreativityBenchmark(),
}
def full_evaluation(self, model):
results = {}
for name, benchmark in self.benchmarks.items():
results[name] = benchmark.evaluate(model)
return results
Alignment Tax Mitigation
Techniques for Reducing Tax
- Multi-task training â Combine alignment with capability training
- Curriculum learning â Align gradually, not all at once
- Regularization â Penalize large deviations from base model
- Selective alignment â Only align harmful behaviors
- Data quality â High-quality alignment data reduces tax
def regularized_dpo(model, preference_data, reference_model, beta=0.1, lambda_reg=0.01):
"""DPO with regularization to minimize capability loss."""
# Standard DPO loss
dpo_loss = compute_dpo_loss(model, preference_data, reference_model, beta)
# Regularization: penalize deviation from reference
reg_loss = 0
for param, ref_param in zip(model.parameters(), reference_model.parameters()):
reg_loss += ((param - ref_param) ** 2).sum()
total_loss = dpo_loss + lambda_reg * reg_loss
return total_loss
Practice Exercises
-
Tax Measurement: Measure the alignment tax of a DPO-aligned model on 5 benchmarks. What is the average tax?
-
Tradeoff Analysis: Plot the helpfulness-harmlessness tradeoff curve by varying the beta parameter. What is the optimal tradeoff point?
-
Selective Alignment: Implement selective alignment that only targets harmful behaviors. Compare the tax to full alignment.
-
Regularization: Test different regularization strengths in DPO. How does regularization affect the alignment tax?
Key Takeaways
What to Learn Next
-> DPO and Preference Optimization Direct preference optimization for alignment.
-> RLHF and Alignment The original RLHF approach.
-> LLM Safety and Red Teaming Safety testing for language models.
-> LLM Evaluation Benchmarks Evaluating LLMs on standard benchmarks.
-> Constitutional AI Using AI feedback for alignment.
-> ML Ethics Ethical considerations in ML systems.