A/B Testing for ML — Complete Guide

A/B testing compares two versions to determine which performs better. Essential for ML model validation.

A/B Testing Framework

1. Hypothesis:
   H₀: No difference between A and B
   H₁: B is better than A

2. Randomization:
   Split users into control (A) and treatment (B)

3. Metrics:
   Primary: Click-through rate, conversion
   Secondary: Revenue, engagement

4. Sample size:
   Power analysis determines needed samples

5. Analysis:
   Statistical test → p-value → Decision

Sample Size Calculation

from statsmodels.stats.power import NormalIndPower

analysis = NormalIndPower()
sample_size = analysis.solve_power(
    effect_size=0.05,  # Minimum detectable effect
    alpha=0.05,         # Significance level
    power=0.80,         # Statistical power
    alternative='larger'
)

Key Takeaways

1. A/B testing validates model improvements in production
2. Random assignment eliminates bias
3. Sample size calculation prevents underpowered tests
4. Statistical significance ≠ practical significance
5. Multi-armed bandits adapt during the test
6. Online ML continuously optimizes
7. Guardrail metrics prevent harm
8. Longer tests capture temporal effects