Why Probability Distributions Matter
A probability distribution is a mathematical function that describes the likelihood of all possible outcomes of a random variable. Distributions come in two families:
- Discrete: Countable outcomes (Bernoulli, Binomial, Poisson, Geometric)
- Continuous: Uncountable outcomes over an interval (Normal, Exponential, Gamma, Beta, Chi-Square, t, F)
Each distribution is defined by its probability density function (PDF) or probability mass function (PMF), its cumulative distribution function (CDF), and a set of parameters that shape its behavior.
Normal Distribution
The Normal distribution is the cornerstone of statistics. Its importance stems from the Central Limit Theorem: the sum of many independent random variables tends toward a Normal distribution, regardless of the original distribution.
Exponential Distribution
The Exponential distribution models the waiting time between consecutive events in a Poisson process. It is the continuous analogue of the Geometric distribution.
Gamma Distribution
The Gamma distribution generalizes the Exponential distribution. While the Exponential models the waiting time for one event, the Gamma models the waiting time for the Ξ±-th event in a Poisson process.
Beta Distribution
The Beta distribution is defined on the interval , making it the natural choice for modeling probabilities and proportions. It is the conjugate prior for the Bernoulli, Binomial, and Negative Binomial likelihoods.
Chi-Square Distribution
The Chi-Square distribution is a special case of the Gamma distribution with shape and rate . It arises naturally as the distribution of sums of squared standard Normal random variables.
t-Distribution (Student's t)
The t-distribution arises when estimating the mean of a normally distributed population with an unknown variance estimated from a small sample. It has heavier tails than the Normal, reflecting the additional uncertainty from estimating .
F-Distribution
The F-distribution is the ratio of two independent Chi-Square random variables, each divided by its degrees of freedom. It is the foundation of ANOVA (Analysis of Variance) and F-tests for comparing model variances.
Distribution Relationships
Python Implementation
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
# --- Normal Distribution ---
mu, sigma = 0, 1
normal = stats.norm(mu, sigma)
print(f"Normal PDF at 0: {normal.pdf(0):.4f}") # 0.3989
print(f"Normal CDF at 1.96: {normal.cdf(1.96):.4f}") # 0.9750
print(f"Normal PPF at 0.975: {normal.ppf(0.975):.4f}") # 1.9600
print(f"Normal mean: {normal.mean():.4f}") # 0.0000
print(f"Normal variance: {normal.var():.4f}") # 1.0000
samples_normal = normal.rvs(size=10000, random_state=42)
# --- Exponential Distribution ---
lam = 2.0
exponential = stats.expon(scale=1/lam)
print(f"Exp PDF at 0.5: {exponential.pdf(0.5):.4f}") # 0.7358
print(f"Exp mean: {exponential.mean():.4f}") # 0.5000
print(f"Exp variance: {exponential.var():.4f}") # 0.2500
samples_exp = exponential.rvs(size=10000, random_state=42)
# --- Gamma Distribution ---
alpha, beta_param = 3.0, 2.0
gamma = stats.gamma(a=alpha, scale=1/beta_param)
print(f"Gamma mean: {gamma.mean():.4f}") # 1.5000
print(f"Gamma variance: {gamma.var():.4f}") # 0.7500
samples_gamma = gamma.rvs(size=10000, random_state=42)
# --- Beta Distribution ---
alpha_b, beta_b = 5.0, 3.0
beta = stats.beta(alpha_b, beta_b)
print(f"Beta mean: {beta.mean():.4f}") # 0.6250
print(f"Beta variance: {beta.var():.4f}") # 0.0268
samples_beta = beta.rvs(size=10000, random_state=42)
# --- Chi-Square Distribution ---
k = 5
chi2 = stats.chi2(df=k)
print(f"Chi2 mean: {chi2.mean():.4f}") # 5.0000
print(f"Chi2 variance: {chi2.var():.4f}") # 10.0000
samples_chi2 = chi2.rvs(size=10000, random_state=42)
# --- t-Distribution ---
nu = 5
t_dist = stats.t(df=nu)
print(f"t mean: {t_dist.mean():.4f}") # 0.0000
print(f"t variance: {t_dist.var():.4f}") # 1.6667
print(f"t 97.5th percentile: {t_dist.ppf(0.975):.4f}") # 2.5706
samples_t = t_dist.rvs(size=10000, random_state=42)
# --- F-Distribution ---
d1, d2 = 5, 10
f_dist = stats.f(dfn=d1, dfd=d2)
print(f"F mean: {f_dist.mean():.4f}") # 1.2500
print(f"F variance: {f_dist.var():.4f}") # 0.9375
samples_f = f_dist.rvs(size=10000, random_state=42)
# --- Visualization ---
fig, axes = plt.subplots(2, 4, figsize=(16, 8))
dists = [
(samples_normal, 'Normal(0,1)', -4, 4),
(samples_exp, 'Exponential(2)', 0, 3),
(samples_gamma, 'Gamma(3,2)', 0, 5),
(samples_beta, 'Beta(5,3)', 0, 1),
(samples_chi2, 'Chi-Square(5)', 0, 15),
(samples_t, 't(5)', -5, 5),
(samples_f, 'F(5,10)', 0, 5),
]
for ax, (data, title, lo, hi) in zip(axes.flat, dists):
ax.hist(data, bins=50, density=True, alpha=0.7, edgecolor='black')
ax.set_title(title)
ax.set_xlim(lo, hi)
axes.flat[-1].axis('off')
plt.tight_layout()
plt.savefig('distributions.png', dpi=150)
plt.show()
# --- Hypothesis Testing Examples ---
# t-test: comparing two sample means
group_a = np.random.normal(loc=100, scale=15, size=50)
group_b = np.random.normal(loc=105, scale=15, size=50)
t_stat, p_value = stats.ttest_ind(group_a, group_b)
print(f"t-statistic: {t_stat:.4f}, p-value: {p_value:.4f}")
# F-test: comparing two variances
f_stat = np.var(group_a, ddof=1) / np.var(group_b, ddof=1)
f_p_value = 1 - stats.f.cdf(f_stat, len(group_a)-1, len(group_b)-1)
print(f"F-statistic: {f_stat:.4f}, p-value: {f_p_value:.4f}")
# Chi-square test: goodness of fit
observed = np.array([30, 50, 20])
expected = np.array([25, 50, 25])
chi2_stat, chi2_p = stats.chisquare(observed, f_exp=expected)
print(f"Chi2-statistic: {chi2_stat:.4f}, p-value: {chi2_p:.4f}")
Applications in AI and Machine Learning
Gaussian Processes
The GP prior is , where is a kernel function. Predictions at new points are obtained by conditioning on observed data β all operations remain in the Normal distribution family.
Reparameterization Trick
Other Applications
Common Mistakes
| Mistake | Why It Is Wrong | Correct Approach |
|---|---|---|
| Assuming all data is Normal | Real data is often skewed or heavy-tailed | Check with Q-Q plots, Shapiro-Wilk test |
| Using Normal for probabilities bounded in [0,1] | Normal assigns probability outside [0,1] | Use Beta distribution |
| Confusing rate and scale parameters | vs | Always check your library's parameterization |
| Ignoring degrees of freedom in t-tests | Small samples need t-distribution, not Normal | Use t for |
| Using z-test when is unknown | Z-test requires known population variance | Use t-test instead |
| Assuming Chi-Square approximation for small expected counts | Chi-square test requires expected counts β₯ 5 | Use Fisher's exact test for small samples |
| Forgetting that F-test is sensitive to non-normality | F-test assumes normally distributed data | Check assumptions or use robust alternatives |
| Using mean and variance for skewed distributions | Mean is misleading for skewed data | Use median and IQR |
| Not distinguishing PMF from PDF | Discrete distributions use PMF, continuous use PDF | Check whether your variable is discrete or continuous |
| Over-interpreting p-values | p < 0.05 does not mean the effect is large or important | Report effect sizes and confidence intervals |