OLS Estimation: From First Principles
Regression Analysis
The Math Behind Regression Coefficients
Ordinary Least Squares finds the coefficient vector that minimizes the sum of squared residuals. Understanding OLS from first principles reveals why regression works and when it breaks down.
-
Data Science — Build foundation for understanding regularization and advanced estimators
-
Econometrics — Derive the Gauss-Markov theorem and BLUE properties
-
Actuarial Science — Implement premium models with transparent coefficient derivation
The normal equations transform data into the best linear unbiased estimates.
Ordinary Least Squares (OLS) is the foundation of linear regression. It finds the coefficient vector that minimizes the sum of squared residuals.
Matrix Formulation
Derivation of the Normal Equations
The Hat Matrix
is an orthogonal projection matrix: and . It projects onto the column space of . The residuals lie in the orthogonal complement.
Properties of OLS Estimators
Estimation of Error Variance
— this is unbiased. The denominator accounts for the parameters estimated.