Regression Analysis
Overview
Simple linear regression models the relationship between one predictor and one outcome: . Multiple linear regression extends this to multiple predictors. Coefficients are estimated via ordinary least squares (OLS), which minimizes the sum of squared residuals. Key diagnostics include checking linearity (residuals vs fitted plot), normality (Q-Q plot of residuals), homoscedasticity (constant residual variance), and independence (no autocorrelation). RΒ² measures the proportion of variance explained; adjusted RΒ² penalizes for adding predictors. Violated assumptions lead to biased coefficients, incorrect standard errors, and invalid inference.
Key Concepts
Diagnostic Checklist
| Assumption | What to Check | How to Check | Remedy if Violated |
|---|---|---|---|
| Linearity | Linear relationship | Residuals vs. fitted plot | Add polynomial terms, transforms |
| Normality | Residuals ~ Normal | Q-Q plot, Shapiro-Wilk | Transform, robust regression |
| Homoscedasticity | Constant variance | Residuals vs. fitted (funnel = bad) | Weighted least squares, robust SE |
| Independence | No autocorrelation | Durbin-Watson test | Time series models, mixed effects |
| No multicollinearity | Predictors not highly correlated | VIF > 10 threshold | Remove/combine predictors, Ridge |
Quick Example
Key Takeaways
Deep Dive
For detailed explanations, worked examples, and Python implementations, explore the dedicated statistics lessons:
Simple Linear Regression
- Simple Linear Regression β Full derivation, OLS, geometric interpretation, and examples
OLS Estimation
- OLS Estimation β Gauss-Markov theorem, BLUE properties, matrix formulation, and efficiency
Assumptions
- Regression Assumptions β Gauss-Markov assumptions, what happens when they fail, and remedies
Diagnostics
- Residual Analysis β Residual plots, Q-Q plots, influence measures, Cook's distance, and leverage
- R-Squared and Adjusted R-Squared β Interpreting model fit, adjusted RΒ², and information criteria
Multiple Regression
- Multiple Linear Regression β Extending to multiple predictors, interpretation, and variable selection
Related Topics
- Multicollinearity β Diagnosing and addressing correlated predictors
- Heteroscedasticity β Non-constant variance and robust standard errors
- Autocorrelation β Serial correlation in time series regression
- Polynomial Regression β Modeling non-linear relationships