Why This Matters
Statistical testing is how we make decisions under uncertainty. Instead of guessing whether a drug works, a feature helps, or a difference is real — we quantify the evidence and let data guide our conclusions.
The Hypothesis Testing Framework
Visual: Rejection Regions
Type I and Type II Errors
The p-value: What It Actually Means
❌ WRONG
"p = 0.03 means there's a 3% chance H₀ is true"
✅ RIGHT
"If H₀ were true, there's a 3% chance of seeing data this extreme"
❌ WRONG
"p < 0.05 means the effect is large"
✅ RIGHT
"p < 0.05 means the effect is unlikely under H₀" (A tiny effect can be significant with large n)
❌ WRONG
"p > 0.05 means no effect exists"
✅ RIGHT
"p > 0.05 means we don't have enough evidence to reject H₀"
One-Sample t-test
Effect Size: Cohen's d
| |d| Value | Interpretation | |-----------|---------------| | 0.2 | Small effect | | 0.5 | Medium effect | | 0.8 | Large effect |
Complete Example
Assumptions
| Assumption | Description | How to Check |
|---|---|---|
| Independence | Observations are independent | Study design, random sampling |
| Normality | Data is approximately normal | Shapiro-Wilk test, Q-Q plot |
| Continuous | Dependent variable is continuous | Data type inspection |
Note: The t-test is robust to mild non-normality for n > 30 due to the Central Limit Theorem.
Two-Sample t-test
Welch's t-test (Unequal Variances)
Paired Samples
Chi-Square Test
Effect Size: Cramér's V
| |V| Value | Interpretation | |----------|---------------| | 0.1 | Small association | | 0.3 | Medium association | | 0.5 | Large association |
ANOVA (Analysis of Variance)
Why Not Use Multiple t-tests?
One-Way ANOVA
Post-Hoc: Tukey's HSD
Non-Parametric Tests
| Parametric | Non-Parametric | When to Use |
|---|---|---|
| One-sample t | Wilcoxon signed-rank | Small sample, non-normal |
| Independent t | Mann-Whitney U | Unequal variances, ordinal data |
| Paired t | Wilcoxon signed-rank (paired) | Paired, non-normal differences |
| One-way ANOVA | Kruskal-Wallis | Non-normal, 3+ groups |
| Pearson r | Spearman rho | Non-linear monotonic relationship |
Multiple Comparisons Problem
Solutions
Power Analysis
Quick Reference: Which Test to Use
Key Takeaways
Practice Exercises
- Drug Trial: Blood pressure in 40 patients after a new drug. Historical mean 120 mmHg. Sample mean 115, std=12. Is the drug effective?
- A/B Test: Website A conversion 12.3% (n=5000), Website B 13.1% (n=5000). Is B significantly better?
- Survey Analysis: Association between education level and preferred news source (n=200). Chi-square test?
- Experiment Design: Detect medium effect (d=0.5) with 90% power. How many subjects per group?