Maximum Likelihood Estimation
Overview
Given data from a distribution , the maximum likelihood estimator finds the parameter value that maximizes the probability of observing the data. The likelihood function is , and we maximize it (or equivalently minimize the negative log-likelihood). For the Normal distribution, MLEs have closed forms: and (biased β uses not ). Fisher information measures how much each observation tells us about , and the CramΓ©r-Rao bound sets a floor on estimator variance: no unbiased estimator can have variance less than .
Key Concepts
MLE for Common Distributions
| Distribution | Parameter | MLE | Notes |
|---|---|---|---|
| Normal | Unbiased | ||
| Normal | Biased (uses not ) | ||
| Poisson | Closed-form | ||
| Bernoulli | Closed-form | ||
| Exponential | Closed-form |
Quick Example
Key Takeaways
Deep Dive
For detailed explanations, worked examples, and Python implementations, explore the dedicated statistics lessons:
Point Estimation
- Point Estimation β MLE, method of moments, and properties of estimators
Properties of Estimators
- Properties of Estimators β Unbiasedness, consistency, efficiency, sufficiency, and the CramΓ©r-Rao bound
Related Topics
- Logistic Regression Statistics β MLE applied to logistic regression
- Confidence Intervals β MLE-based Wald intervals
- Simple Linear Regression β OLS as a special case of MLE under Normal errors
- Bayesian Regression β Bayesian alternative to MLE estimation