Permutation Tests
Hypothesis Testing
Let the Data Generate the Null Distribution
Permutation tests create null distributions by rearranging the data itself, making no assumptions about population distributions. They provide exact p-values and work with any test statistic.
- Non-Normal Data — Testing hypotheses when distributional assumptions are violated
- Small Samples — Obtaining valid inference with limited observations
- Complex Statistics — Testing custom statistics that lack standard distributional theory
Permutation tests are the ultimate fallback when standard methods fail.
How They Work
- Compute the observed test statistic
- Repeatedly shuffle (permute) group labels
- Compute the test statistic for each permutation
- P-value = proportion of permuted statistics as extreme as the observed
Python Implementation
Visualization
# Visualize permutation distribution
fig, ax = plt.subplots(figsize=(10, 5))
ax.hist(perm_diffs, bins=50, density=True, color='lightblue', edgecolor='black', alpha=0.7)
ax.axvline(observed_diff, color='red', linewidth=2, linestyle='--',
label=f'Observed = {observed_diff:.3f}')
ax.axvline(-observed_diff, color='red', linewidth=2, linestyle='--')
ax.fill_betweenx([0, 0.15], -20, -abs(observed_diff), alpha=0.3, color='red', label=f'p = {p_perm:.4f}')
ax.fill_betweenx([0, 0.15], abs(observed_diff), 20, alpha=0.3, color='red')
ax.set_title(f'Permutation Distribution of Difference in Means\n(n_perm={n_permutations:,})')
ax.set_xlabel('Difference in Means (A - B)')
ax.legend()
plt.tight_layout()
plt.savefig('permutation_test.png', dpi=150)
plt.show()