Linear Regression: Math, Code and Assumptions
The Foundation of Machine Learning
Linear regression is the most fundamental algorithm in ML. Despite its simplicity, understanding it deeply provides insight into all supervised learning methods.
ML Algorithm Landscape
1. Simple Linear Regression
Mathematical Formulation
Model:
Where:
- = intercept (bias) — value of when
- = slope (weight) — change in for unit change in
- = error term —
How this diagram works: This diagram shows the core concept of simple linear regression — fitting a straight line through scattered data points to model the relationship between a feature (x) and a target (y). The blue data points represent actual observations, while the purple regression line represents the model's predictions. The red dashed lines (residuals) show the vertical distance between each data point and the line, representing prediction errors. The goal of linear regression is to minimize these residuals by finding the optimal intercept (β₀) and slope (β₁) that produce the smallest total squared error.
2. Cost Function (Ordinary Least Squares)
Mean Squared Error (MSE):
Goal: Find that minimize
Closed-Form Solution (Normal Equation):
3. Gradient Descent
Update Rule:
Partial Derivatives:
Where = learning rate (step size)
4. Multiple Linear Regression
Model:
Matrix Form:
Where (design matrix with intercept column)
Normal Equation (Matrix):
5. Model Evaluation Metrics
R² Score (Coefficient of Determination):
- : Perfect fit
- : Model predicts the mean
- : Model is worse than predicting the mean
Adjusted R²:
6. Assumptions of Linear Regression
Checking Assumptions with Residual Plots
7. Implementation in Python
Key Takeaways
- Linear regression finds the best-fit line through data points
- Cost function (MSE) measures prediction error — minimize it
- Gradient descent iteratively updates weights to find minimum
- R² score tells you how much variance the model explains
- Validate assumptions before trusting the model
- Regularization (Ridge/Lasso) prevents overfitting
Next: Logistic Regression
Extend linear regression to classification with the sigmoid function.