Bayesian Statistics

Overview

Bayesian inference updates prior beliefs about parameters using observed data via Bayes' rule: posterior ∝ likelihood × prior. The prior encodes beliefs before seeing data. The likelihood is the probability of the data given the parameters. The posterior is the updated belief after seeing data. Conjugate priors (e.g., Beta-Binomial, Normal-Normal) yield closed-form posteriors for exact analytical updates. MAP estimation finds the mode of the posterior, equivalent to MLE with regularization. For complex models, MCMC methods (Gibbs sampling, HMC) sample from the posterior distribution numerically.

Key Concepts

Conjugate Prior Families

Likelihood	Prior	Posterior	Use Case
Bernoulli/Binomial	Beta	Beta	Proportions, click rates
Normal (known )	Normal	Normal	Mean estimation
Poisson	Gamma	Gamma	Count data
Normal (unknown , )	Normal-Inverse-Gamma	Normal-Inverse-Gamma	Full normal model

Prior Strength Effects

Prior Strength	Effect on Posterior	When to Use
Weak (large )	Posterior dominated by data	Large samples, little prior knowledge
Strong (small )	Posterior dominated by prior	Small samples, strong prior knowledge
Flat (uniform)	Posterior = likelihood (up to constant)	Non-informative analysis

Quick Example

Key Takeaways

Deep Dive

For detailed explanations, worked examples, and Python implementations, explore the dedicated statistics lessons:

Bayesian Regression

Bayesian Regression — Full Bayesian treatment of regression with posterior distributions over coefficients

Hierarchical Models

Hierarchical Bayesian — Multi-level models with partial pooling, random effects, and shrinkage

MCMC Diagnostics

MCMC Diagnostics — Convergence checks, trace plots, effective sample size, statistic, and autocorrelation

Bayesian Statistics