LLM Evaluation
Bias and Fairness in LLMs
LLMs inherit and amplify societal biases present in their training data. Understanding, measuring, and mitigating these biases is essential for responsible deployment.
- Sources — Training data, annotation, algorithmic amplification
- Measurement — StereoSet, CrowS-Pairs, BBQ benchmarks
- Mitigation — Data curation, fine-tuning, prompting, and decoding
Fairness is not an act of kindness—it is an act of justice.
Bias and Fairness in LLMs
LLMs inherit and amplify societal biases present in their training data. Understanding, measuring, and mitigating these biases is essential for responsible deployment across all applications.
Sources of Bias
1. Training Data Bias
The most fundamental source: the training corpus reflects historical and societal biases.
2. Annotation Bias
Human annotators bring their own biases when creating training data for instruction tuning or RLHF.
3. Algorithmic Amplification
Training objectives and architectural choices can amplify small data biases into large output biases.
Types of Bias in Detail
Understanding the different forms bias takes is essential for effective mitigation:
How Bias Manifests in LLMs
Bias can appear at multiple stages of the LLM pipeline:
- Data collection: Web crawls overrepresent certain demographics and viewpoints
- Pretraining: The model learns statistical associations from biased data
- Alignment/RLHF: Human raters may have systematic biases in preference data
- Deployment: Users may use outputs in ways that amplify existing biases
Measuring Bias
Benchmark-Based Evaluation
Several standardized benchmarks exist for measuring social bias:
| Benchmark | Approach | Bias Categories | Method |
|---|---|---|---|
| StereoSet | Sentence completion | Gender, race, religion, profession | Probability comparison |
| CrowS-Pairs | Sentence pairs | 9 categories | Relative likelihood |
| BBQ | Question answering | Ambiguous/disambiguated contexts | Accuracy parity |
| WinoBias | Coreference | Gender, occupation | Winograd schema |
| BBQ-Civil | QA | Intersectional | Context manipulation |
Counterfactual Evaluation
The counterfactual method measures bias by comparing outputs when demographic attributes are swapped:
Debiasing Techniques
Data-Level Debiasing
- Resampling: Balance representation of demographic groups
- Data augmentation: Generate counterfactual examples
- Toxicity filtering: Remove explicitly biased content
Training-Level Debiasing
The adversarial component encourages the model to produce outputs from which the demographic attribute cannot be predicted—enforcing demographic parity.
Inference-Level Debiasing
- Prompt-based debiasing: Instruct the model to be fair and unbiased
- Constrained decoding: Block stereotypical token sequences
- Output filtering: Post-hoc filtering of biased outputs
Fairness Metrics
| Metric | Definition | Property |
|---|---|---|
| Demographic Parity | P(ŷ=1 | g=0) = P(ŷ=1 | g=1) | Independence |
| Equalized Odds | P(ŷ=1 | g, y=1) equal across g | Separation |
| Predictive Parity | P(y=1 | ŷ=1, g) equal across g | Sufficiency |
| Counterfactual Fairness | P(ŷ | do(g=0)) = P(ŷ | do(g=1)) | Invariance |
Practice Exercises
-
Conceptual: Explain why equalized odds and demographic parity can conflict. Give a concrete example where satisfying one necessarily violates the other.
-
Mathematical: Given a model that predicts "doctor" with probability 0.8 for men and 0.3 for women in a template completion task, compute the counterfactual token bias. Is this model biased?
-
Practical: Using the StereoSet dataset, evaluate a small LLM's implicit bias across the four categories. Which category shows the most bias?
-
Research: Compare the effectiveness of data-level debiasing vs. inference-level debiasing. What are the tradeoffs in terms of bias reduction and capability preservation?
What to Learn Next
-> Hallucination Detection and Mitigation Detecting and reducing factual errors in LLM outputs.
-> LLM Safety and Red Teaming Systematic adversarial testing for safety vulnerabilities.
-> Constitutional AI Training models with explicit behavioral principles.
-> RLHF and Alignment Aligning language models with human preferences and values.
-> LLM Benchmarking Suites Comprehensive benchmarks including bias evaluation.
-> Copyright and Legal Issues Legal frameworks governing AI fairness and discrimination.