Simple Linear Regression
Regression Analysis
Modeling the Relationship Between Two Variables
Simple linear regression quantifies how one variable changes with another, forming the foundation of predictive modeling. It estimates the line that best fits the data by minimizing squared residuals.
-
Economics — Predict consumer spending based on income levels
-
Healthcare — Model the relationship between dosage and patient response
-
Engineering — Relate temperature to material expansion coefficients
Every complex model begins with understanding a single straight line.
Simple linear regression models the linear relationship between a predictor variable and a response variable .
The Statistical Model
Ordinary Least Squares (OLS) Estimation
OLS finds the estimates that minimize the sum of squared residuals:
Properties of OLS Estimators
The Coefficient of Determination ()
. An means 75% of the variability in is explained by the linear relationship with .
The Four Regression Assumptions (LINE)
| Assumption | Mathematical Statement | Diagnostic Check |
|-----------|----------------------|------------------|
| Linearity | | Scatter plot; residual vs. fitted plot |
| Independence | for | Study design; Durbin–Watson test |
| Normality | | Q–Q plot; Shapiro–Wilk test |
| Equal variance | (constant) | Residual vs. fitted plot; Breusch–Pagan test |
Hypothesis Tests for the Slope
The -test with is equivalent to the -test in simple regression: .
Prediction Intervals
For a new observation at , the prediction interval is wider than the confidence interval for the mean:
The under the square root makes the prediction interval always wider than the confidence interval for the mean response.