Maximum Likelihood Estimation
Overview
Given data from a distribution , the maximum likelihood estimator finds the parameter value that maximizes the probability of observing the data. The likelihood function is , and we maximize it (or equivalently minimize the negative log-likelihood). For the Normal distribution, MLEs have closed forms: and (biased — uses not ). Fisher information measures how much each observation tells us about , and the Cramér-Rao bound sets a floor on estimator variance: no unbiased estimator can have variance less than .
Key Concepts
MLE for Common Distributions
| Distribution | Parameter | MLE | Notes |
|---|---|---|---|
| Normal | Unbiased | ||
| Normal | Biased (uses not ) | ||
| Poisson | Closed-form | ||
| Bernoulli | Closed-form | ||
| Exponential | Closed-form |
Quick Example
Key Takeaways
Deep Dive
For detailed explanations, worked examples, and Python implementations, explore the dedicated statistics lessons:
Point Estimation
- Point Estimation — MLE, method of moments, and properties of estimators
Properties of Estimators
- Properties of Estimators — Unbiasedness, consistency, efficiency, sufficiency, and the Cramér-Rao bound
Related Topics
- Logistic Regression Statistics — MLE applied to logistic regression
- Confidence Intervals — MLE-based Wald intervals
- Simple Linear Regression — OLS as a special case of MLE under Normal errors
- Bayesian Regression — Bayesian alternative to MLE estimation