Hypothesis Testing

Overview

Every hypothesis test begins by formulating two competing statements about a population parameter. The null hypothesis () is the default assumption of no effect. The alternative hypothesis () is the claim that an effect exists. A test statistic measures how far observed data deviates from . The p-value quantifies the probability of seeing results at least as extreme if is true. We reject when the p-value falls below the significance level . Two types of errors are possible: Type I (false positive, probability ) and Type II (false negative, probability ). Power () is the probability of detecting a real effect, and increases with effect size, sample size, and .

Key Concepts

Error Matrix

	is True	is False
Reject	Type I Error () — false positive	Power () — true positive
Fail to Reject	Correct — true negative	Type II Error () — false negative

Effect Size Benchmarks (Cohen's d)

Effect	Cohen's d	Interpretation
Small	0.2	Subtle, hard to detect
Medium	0.5	Noticeable practical effect
Large	0.8	Strong, clearly visible

P-Value Interpretation

P-Value	Evidence Against
	Very strong
	Strong
	Weak
	Little or none

Quick Example

Key Takeaways

Deep Dive

For detailed explanations, worked examples, and Python implementations, explore the dedicated statistics lessons:

Hypothesis Formulation

Null and Alternative Hypothesis — How to formulate and , one-sided vs. two-sided, and common patterns

Errors and Significance

Type I and Type II Errors — Error matrix, trade-off, and real-world consequences
P-Values — Calculation, interpretation, and common misinterpretations
Significance Levels — Choosing , multiple testing, and when to use 0.01 vs 0.05

Power and Effect Size

Power of a Test — Factors affecting power, a priori power analysis, and underpowered studies
Effect Size — Cohen's d, Hedges' g, eta-squared, and why practical significance matters

Hypothesis Testing