ANOVA

ℹ️ Why It Matters

When comparing means across three or more groups, running multiple t-tests inflates the Type I error rate. With 3 groups, there are 3 pairwise comparisons, each at α = 0.05, giving a family-wise error rate of about 14%. ANOVA (Analysis of Variance) tests whether any group means differ while controlling the overall error rate at α. It is the standard method for experiments with multiple treatment conditions, A/B/n testing, and any scenario involving categorical predictors with continuous outcomes.

Overview

ANOVA partitions total variance in the data into between-group variance (differences attributable to the treatment) and within-group variance (random noise within groups). The F-statistic is the ratio of between-group to within-group variance: $F = MS_{between} / MS_{within}$ . A large F indicates that group means are more spread than expected by chance. One-way ANOVA tests means across levels of a single factor. Two-way ANOVA tests two factors and their interaction. ANOVA is an omnibus test — it only tells you that at least one group differs, not which groups differ. Post-hoc tests (Tukey's HSD, Bonferroni) identify specific pairwise differences after a significant F-test.

Key Concepts

F-Statistic (One-Way ANOVA)

F = \frac{\text{MS}_{between}}{\text{MS}_{within}} = \frac{SS_{between} / (k-1)}{SS_{within} / (N-k)}

Here,

$SS_{between}$ =Sum of squares between groups: $\sum n_j(\bar{x}_j - \bar{x})^2$
$SS_{within}$ =Sum of squares within groups: $\sum\sum(x_{ij} - \bar{x}_j)^2$
$k$ =Number of groups
$N$ =Total number of observations

F-Distribution

F \sim F_{k-1, N-k}

Here,

$k-1$ =Numerator degrees of freedom (between groups)
$N-k$ =Denominator degrees of freedom (within groups)

Effect Size: Eta-Squared

\eta^2 = \frac{SS_{between}}{SS_{total}}

Here,

$SS_{total}$ =Total sum of squares: $SS_{between} + SS_{within}$

Two-Way ANOVA Model

y_{ijk} = \mu + \alpha_i + \beta_j + (\alpha\beta)_{ij} + \epsilon_{ijk}

Here,

$\alpha_i$ =Main effect of factor A
$\beta_j$ =Main effect of factor B
$(\alpha\beta)_{ij}$ =Interaction effect between factors A and B
$\epsilon_{ijk}$ =Random error term

ANOVA Assumptions

Independence: Observations are independent within and across groups. Violations (e.g., repeated measures) require different tests.
Normality: Residuals are approximately normally distributed. Robust to violations with large $n$ (CLT).
Homogeneity of variances: Groups have equal population variances. Test with Levene's test. If violated, use Welch's ANOVA.

Sum of Squares Decomposition

SS_{total} = SS_{between} + SS_{within}

$SS_{between}$ : Variation due to group differences (explainable)
$SS_{within}$ : Variation within groups (unexplainable noise)
$\eta^2 = SS_{between} / SS_{total}$ : Proportion of total variance explained by groups

Post-Hoc Tests

Test	When to Use	Controls	Conservativeness
Tukey's HSD	All pairwise comparisons	Family-wise error	Moderate
Bonferroni	Few planned comparisons	Family-wise error	Conservative
Scheffe	All possible contrasts	Family-wise error	Most conservative
Dunnett	Each group vs. control	Family-wise error	Moderate

Quick Example

📝One-Way ANOVA

Three drugs tested: $F = 4.5$ , $df_1 = 2$ , $df_2 = 87$ .

Using the F-distribution: $p \approx 0.013 < 0.05$ . Reject $H_0$ : at least one drug differs.

ANOVA is omnibus — we know something differs, but not what. Post-hoc Tukey HSD identifies which specific pairs differ. Without post-hoc, we cannot claim Drug A is better than Drug B.

📝Two-Way ANOVA

Testing effect of drug (A vs B) and dosage (low vs high) on recovery time. Two-way ANOVA tests three hypotheses simultaneously:

Does drug type matter? (main effect of drug)
Does dosage matter? (main effect of dosage)
Does the effect of drug depend on dosage? (interaction)

If the interaction is significant, the effect of drug depends on dosage level — you cannot interpret main effects in isolation.

📝Post-Hoc Analysis

ANOVA with 4 groups yields $F = 5.2$ , $p = 0.003$ . Reject $H_0$ : at least one group differs.

Tukey's HSD reveals: Group A vs C ( $p = 0.001$ ), Group A vs D ( $p = 0.02$ ), Group B vs C ( $p = 0.004$ ), Group B vs D ( $p = 0.03$ ). Groups C and D don't differ from each other ( $p = 0.91$ ).

Assumption Checking in Python

from scipy import stats

# Levene's test for equal variances
_, p_levene = stats.levene(group1, group2, group3)
print(f"Levene's test p-value: {p_levene:.3f}")

# Shapiro-Wilk test for normality of residuals
_, p_normal = stats.shapiro(residuals)
print(f"Normality test p-value: {p_normal:.3f}")

If Levene's test is significant ( $p < 0.05$ ), use Welch's ANOVA or the nonparametric Kruskal-Wallis test instead.

Key Takeaways

📋Summary: ANOVA

Purpose: Compare means across 3+ groups while controlling family-wise Type I error.
F-statistic: Between-group variance / Within-group variance. Large F → reject $H_0$ .
Hypotheses: $H_0$ : all group means equal. $H_1$ : at least one differs.
Post-hoc: ANOVA is omnibus — use Tukey's HSD or Bonferroni to find which groups differ.
Assumptions: Normality, equal variances, independence. Levene's test checks homoscedasticity.
Effect Size: $\eta^2$ = proportion of variance explained by the group factor.
Relationship to t-test: One-way ANOVA with 2 groups is equivalent to a two-sample t-test ( $F = t^2$ ).
Two-Way ANOVA: Tests main effects of two factors and their interaction term.
Nonparametric Alternative: Kruskal-Wallis test when ANOVA assumptions are violated.

Deep Dive

For detailed explanations, worked examples, and Python implementations, explore the dedicated statistics lessons:

ANOVA

One-Way ANOVA — Complete derivation, F-distribution, sum of squares decomposition, post-hoc tests, and Python implementation

Related Tests

F-Test for Equality of Variances — Testing whether two groups have equal variances using the F-distribution
Levene's Test — Robust test for homogeneity of variances when normality is questionable
Two-Sample t-Test — ANOVA with 2 groups is equivalent to a t-test ( $F = t^2$ )

ANOVA

ANOVA

Overview

Key Concepts

F-Statistic (One-Way ANOVA)

F-Distribution

Effect Size: Eta-Squared

Two-Way ANOVA Model

ANOVA Assumptions

Sum of Squares Decomposition

Post-Hoc Tests

Quick Example

📝One-Way ANOVA

📝Two-Way ANOVA

📝Post-Hoc Analysis

Assumption Checking in Python

Key Takeaways

📋Summary: ANOVA

Deep Dive

ANOVA

Related Tests

Related Topics