Bias and Fairness in Healthcare AI

What is Bias in Healthcare AI?

Bias in healthcare AI refers to systematic errors that produce unfair outcomes across different patient populations, leading to misdiagnosis, delayed treatment, and exacerbation of existing health disparities. These biases are not merely theoretical concerns—they translate directly into clinical harm. A dermatology AI trained predominantly on lighter skin tones misdiagnoses melanoma in darker-skinned patients at rates 15-20% higher than in lighter-skinned patients, contributing to the 25% higher melanoma mortality observed in Black patients despite lower incidence. Similarly, pulse oximeters—used universally for oxygen saturation monitoring—demonstrate 3x greater failure rates in Black patients due to melanin interference with light absorption, yet AI models trained on this data perpetuate the same measurement bias into automated clinical decision support systems. The sources of bias are multifactorial, spanning data collection, labeling, model training, and deployment stages, requiring systematic identification and mitigation at each point in the AI development pipeline.

The clinical impact of biased AI is amplified by the feedback loops inherent in healthcare systems. When an AI model underperforms for a specific demographic group, clinicians learn to distrust the model's predictions for those patients, reverting to manual assessment. This selective deployment creates a situation where the group most in need of AI assistance receives the least benefit, widening rather than narrowing health disparities. Furthermore, biased models can encode and perpetuate historical inequities in clinical practice—if training labels reflect historical underdiagnosis or undertreatment of certain conditions in minority populations, the AI learns to reproduce these patterns as if they were medically appropriate rather than reflecting systemic failures in healthcare access and delivery.

Addressing healthcare AI bias requires a comprehensive approach that recognizes bias as a sociotechnical problem requiring both technical and organizational solutions. Technical interventions span the entire ML pipeline: pre-processing approaches reweight or resample training data to address representation imbalances; in-processing methods incorporate fairness constraints directly into model training objectives; and post-processing techniques adjust model outputs to achieve desired fairness properties. However, technical solutions alone are insufficient—organizational interventions including diverse development teams, community engagement, transparent reporting of demographic performance, and ongoing post-deployment monitoring are equally essential for ensuring that AI systems serve all patient populations equitably.

The regulatory landscape is evolving to mandate fairness requirements for healthcare AI, with the EU AI Act classifying medical AI as high-risk and requiring conformity assessments that include fairness evaluation. The FDA is developing guidance for evaluating demographic subgroup performance in AI/ML-based medical devices, requiring sponsors to report sensitivity, specificity, and AUC stratified by age, sex, race, and ethnicity. These regulatory requirements create accountability mechanisms that incentivize fairness-aware AI development, though significant challenges remain in defining appropriate fairness thresholds for different clinical applications and in developing standardized evaluation protocols that capture intersectional biases affecting patients at the intersection of multiple protected attributes.

Key Sources

• Data bias: Training datasets underrepresent minority populations, creating models that generalize poorly to underrepresented groups
• Label bias: Historical diagnostic patterns encode existing inequities, teaching models to reproduce past discrimination
• Measurement bias: Clinical instruments perform differently across skin tones, body habitus, and physiological differences
• Deployment bias: Models trained at academic centers fail in community settings with different patient populations and equipment
• Selection bias: Clinical trial populations do not reflect real-world demographic diversity

Fairness Metrics

Demographic Parity

Where each parameter means:

• — model's binary prediction (1 = positive, 0 = negative)
• — protected attribute (0 = reference group, 1 = disadvantaged group), such as race, sex, or age
• — positive prediction rate (selection rate) for the reference group
• — positive prediction rate for the disadvantaged group
• Intuition: Demographic parity requires that the model selects individuals at equal rates across groups, regardless of base rate differences. If a screening AI refers 20% of White patients for further evaluation, it should also refer 20% of Black patients. This metric does not account for differences in true disease prevalence between groups, making it appropriate for settings where equal opportunity for screening is prioritized over diagnostic accuracy

Equalized Odds

Where each parameter means:

• — true label (0 = negative, 1 = positive)
• — model's prediction
• — protected attribute
• — true positive rate (sensitivity) for the reference group
• — true positive rate (sensitivity) for the disadvantaged group
• — false positive rate for the reference group
• — false positive rate for the disadvantaged group
• Intuition: Equalized odds requires that the model achieves equal true positive rates AND equal false positive rates across groups. This is clinically desirable because it means the model is equally accurate for all groups—patients with the disease have equal probability of detection regardless of demographics, and healthy patients have equal probability of avoiding unnecessary procedures

Calibration

Where each parameter means:

• — model's predicted probability (confidence score)
• — observed disease prevalence among reference group patients given predicted probability
• — observed disease prevalence among disadvantaged group patients given predicted probability
• — the predicted probability itself (the target for perfect calibration)
• Intuition: Calibration requires that a prediction of 70% risk means the same thing regardless of patient group—if the model predicts 70% cancer risk for a White patient and a Black patient, both should have approximately 70% true cancer prevalence. Miscalibration by group leads to systematic over- or under-treatment for specific populations

import torch
import numpy as np

class FairnessMetrics:
    def __init__(self, predictions, labels, sensitive_attr):
        self.preds = predictions
        self.labels = labels
        self.groups = sensitive_attr

    def demographic_parity(self):
        rate_0 = self.preds[self.groups == 0].mean()
        rate_1 = self.preds[self.groups == 1].mean()
        return abs(rate_0 - rate_1)

    def equalized_odds(self):
        tpr_0 = self._tpr(self.groups == 0)
        tpr_1 = self._tpr(self.groups == 1)
        fpr_0 = self._fpr(self.groups == 0)
        fpr_1 = self._fpr(self.groups == 1)
        return abs(tpr_0 - tpr_1) + abs(fpr_0 - fpr_1)

    def _tpr(self, mask):
        pos = self.labels[mask] == 1
        return self.preds[mask][pos].mean() if pos.sum() > 0 else 0

    def _fpr(self, mask):
        neg = self.labels[mask] == 0
        return self.preds[mask][neg].mean() if neg.sum() > 0 else 0

    def calibration_gap(self, n_bins=10):
        gaps = []
        for i in range(n_bins):
            low, high = i / n_bins, (i + 1) / n_bins
            mask_0 = (self.groups == 0) & (self.preds >= low) & (self.preds < high)
            mask_1 = (self.groups == 1) & (self.preds >= low) & (self.preds < high)
            if mask_0.sum() > 0 and mask_1.sum() > 0:
                gap = abs(self.labels[mask_0].mean() - self.labels[mask_1].mean())
                gaps.append(gap)
        return np.mean(gaps) if gaps else 0

preds = torch.randint(0, 2, (1000,)).float()
labels = torch.randint(0, 2, (1000,)).float()
groups = torch.bernoulli(torch.ones(1000) * 0.5).int()

metrics = FairnessMetrics(preds, labels, groups)
print(f"Demographic parity gap: {metrics.demographic_parity():.3f}")
print(f"Equalized odds gap: {metrics.equalized_odds():.3f}")

Bias Mitigation Strategies

Pre-processing: Reweighting

Where each parameter means:

• — weight assigned to training sample , adjusting its contribution to the loss function
• — marginal probability of the label in the entire training distribution
• — marginal probability of group in the entire training distribution
• — conditional probability of label within group , which may differ from the marginal due to representation imbalances
• Intuition: Reweighting upweights samples from underrepresented (group, label) combinations and downweights overrepresented ones, ensuring that each combination contributes equally to the loss. If only 5% of training data is from Black patients with melanoma, but they represent 15% of melanoma cases in the population, reweighting increases their influence by a factor of 3 to match their true clinical prevalence

In-processing: Adversarial Debiasing

Where each parameter means:

• — task loss (e.g., cross-entropy for cancer detection) that the predictor model (parameterized by ) aims to minimize
• — adversarial loss that the adversary (parameterized by ) aims to maximize, predicting the protected attribute from the predictor's learned representations
• — fairness regularization strength controlling the tradeoff between task performance and fairness; larger values enforce stronger fairness constraints but may reduce accuracy
• Intuition: The predictor learns representations that are useful for the clinical task but uninformative about the protected attribute. The adversary tries to predict race/sex from these representations, and the predictor is penalized when the adversary succeeds. At equilibrium, the predictor's representations contain no information about the protected attribute, achieving fair predictions while maintaining clinical utility

Comparison of Mitigation Methods

Real-World Case Study: Removing Race from eGFR

The estimated glomerular filtration rate (eGFR), used to assess kidney function and determine transplant eligibility, historically included a race-based multiplier that overestimated kidney function in Black patients by 10-15%. A 2021 multi-center study demonstrated that removing the race variable from the eGFR equation reclassified 33% of Black patients from CKD stage 3 (mild dysfunction) to stage 4 (severe dysfunction), qualifying them for nephrology referral and transplant waitlisting. When AI models trained on the race-adjusted eGFR were evaluated for fairness, the calibration gap between Black and White patients was 0.18 (p<0.001), meaning the model's risk predictions were systematically less accurate for Black patients. After retraining with the race-neutral equation and fairness constraints (adversarial debiasing with ), the calibration gap reduced to 0.03 with only 1.2% AUC reduction, demonstrating that fairness improvements need not come at the cost of substantial performance degradation.

Common Challenges

• Fairness-accuracy tradeoff: Increasing fairness constraints may reduce overall accuracy by 1-5%, requiring careful calibration of the fairness-utility balance based on clinical application and patient population characteristics
• Intersectional bias: Multiple protected attributes interact in complex ways (e.g., elderly Black women may face compounded disparities not captured by single-attribute analysis), requiring intersectional fairness evaluation across all demographic subgroups
• Proxy variables: Socioeconomic status, zip code, and insurance type can proxy race indirectly, enabling biased predictions even when race is explicitly excluded from the model
• Legal constraints: Fairness definitions may conflict with legal requirements (e.g., equalized odds may require different treatment thresholds that violate anti-discrimination laws), necessitating careful legal-ethical analysis
• Dynamic populations: Patient demographics shift over time due to migration, aging, and population changes, requiring continuous fairness monitoring and model retraining to maintain equitable performance

Summary

Healthcare AI bias arises from data, labels, measurement, and deployment factors that systematically disadvantage specific patient populations. Fairness metrics—demographic parity, equalized odds, and calibration—quantify disparities while mitigation strategies at pre-processing, in-processing, and post-processing stages can reduce bias. The clinical evidence demonstrates that fairness improvements need not come at substantial performance cost, with adversarial debiasing achieving 15-30% fairness improvement with only 1-5% accuracy reduction across multiple healthcare applications.