P-Values: The Most Misunderstood Number in Statistics
Hypothesis Testing
What P-Values Actually Tell You
The p-value is the probability of observing data as extreme as yours if the null hypothesis were true — not the probability that H₀ is true. Correct interpretation prevents the most common statistical mistakes.
- Scientific Publishing — Understanding why p < 0.05 is not a stamp of truth
- Business Decisions — Knowing when statistical significance matters versus practical significance
- Legal Settings — Interpreting statistical evidence in forensic and discrimination cases
The p-value answers a specific question — make sure you're asking the right one.
The p-value is simultaneously the most used and most misused concept in statistics. Its correct interpretation requires careful attention to conditional probability.
The Exact Definition
Formal Framework
What a P-Value IS and IS NOT
| Statement | Correct? | Why |
|---|---|---|
| "p = 0.03 means there's a 3% chance is true" | WRONG | P-value is conditional on , not |
| "p = 0.03 means: if were true, only 3% of samples would yield this extreme a result" | CORRECT | This is the definition |
| "p = 0.03 means the result is practically important" | WRONG | Statistical significance ≠ practical significance |
| "p = 0.03 means the study will replicate 97% of the time" | WRONG | Replication probability depends on true effect size |
| "p > 0.05 proves is true" | WRONG | Failure to reject ≠ acceptance of |
The P-Value Confounds Effect Size with Sample Size
Implication: A tiny, practically meaningless effect will be "statistically significant" with a large enough sample. Conversely, a large effect may not reach significance with a small sample.
The ASA Statement on P-Values (2016)
The American Statistical Association issued six principles:
- P-values can indicate how incompatible the data are with .
- P-values do not measure the probability that is true.
- Scientific conclusions should not be based on whether a p-value exceeds a threshold.
- Proper inference requires full reporting and transparency.
- A p-value does not measure the size or importance of an effect.
- By itself, a p-value does not provide a good measure of evidence.
Recommended Reporting Practice
P-Value and Confidence Intervals
This equivalence means confidence intervals contain strictly more information than p-values: they convey both the direction and magnitude of the effect, not just whether it differs from zero.