From Scatter Plots to Predictions — The Simplest ML Algorithm
Linear regression finds the best straight line through your data. It is fast, interpretable, and a powerful baseline for any regression problem.
-
Ordinary Least Squares — The closed-form solution for optimal parameters
-
Gradient Descent — The iterative optimization approach that scales
-
Evaluation Metrics — R², MSE, and MAE for measuring performance
"All models are wrong, but some are useful." — George Box
Prerequisites
Before diving in, make sure you're comfortable with:
- Basic Python — Variables, loops, functions, and NumPy
- Vectors and Matrices — Dot product, matrix multiplication (see Math Foundations)
- Derivatives — Partial derivatives, gradients (see Math Foundations)
- Basic Statistics — Mean, variance, correlation
Learning Objectives
After completing this tutorial, you will be able to:
- Explain how linear regression finds the best-fitting line
- Implement both OLS (closed-form) and gradient descent solutions
- Compute and interpret R², MSE, RMSE, and MAE
- Check the five key assumptions of linear regression
- Extend simple linear regression to multiple and polynomial regression
- Apply regularization (Ridge, Lasso) to prevent overfitting
Linear Regression — Complete Guide
Linear regression is the simplest and most fundamental ML algorithm. It models the relationship between variables as a straight line.
Simple Linear Regression
Finding the Best Line
Ordinary Least Squares (OLS)
Gradient Descent
Cost Function Surface and Gradient Descent Path
Multiple Linear Regression
Evaluation Metrics
Assumptions
Polynomial Regression
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)
model = LinearRegression().fit(X_poly, y)
Real-World Applications
Real Estate Pricing
Zillow's Zestimate uses linear regression as a baseline model for home valuations. Features include square footage, number of bedrooms, location (latitude/longitude), and neighborhood statistics. Even complex deep learning models often start with linear regression as a benchmark.
Financial Forecasting
Stock analysts use multiple linear regression to model relationships between stock prices and factors like interest rates, company earnings, and market indices. The interpretability of coefficients (each factor's contribution) makes it valuable for risk analysis.
Medical Research
Clinical trials use linear regression to model the relationship between drug dosage and patient outcomes, controlling for age, weight, and other covariates. The assumptions of linear regression (normality, homoscedasticity) are critical for valid statistical inference.
Energy Consumption
Electric utilities predict demand using regression models with features like temperature, time of day, day of week, and economic indicators. These predictions drive power generation scheduling and grid management decisions.
Sports Analytics
Moneyball-style analysis uses regression to estimate player value based on statistics like batting average, on-base percentage, and defensive metrics. Linear regression reveals which statistics most strongly predict team success.
Common Mistakes & How to Avoid Them
Mistake 1: Assuming linearity without checking
- Problem: Fitting a linear model to nonlinear data produces poor predictions
- Solution: Plot residuals vs. predicted values. Random scatter = linear is OK. Patterns (curves, funnels) = linear is wrong.
Mistake 2: Ignoring multicollinearity
- Problem: Highly correlated features (e.g., height in cm and height in inches) make coefficient estimates unstable
- Solution: Check VIF (Variance Inflation Factor). VIF > 10 indicates problematic multicollinearity. Remove or combine correlated features.
Mistake 3: Not scaling features before regularization
- Problem: L1/L2 regularization penalizes coefficient magnitude. Features with larger scales get unfairly penalized
- Solution: Always standardize features before applying Ridge or Lasso regression.
Mistake 4: Extrapolating beyond training data
- Problem: Linear regression assumes the relationship continues linearly outside the observed range
- Solution: Be cautious with predictions far from training data. The model may not generalize.
Mistake 5: Using R² as the only metric
- Problem: High R² doesn't mean the model is good — it could overfit, have non-normal residuals, or violate assumptions
- Solution: Always check residual plots, p-values, and confidence intervals alongside R².
Interview Questions
Q1: What is the difference between R² and adjusted R²? A: R² always increases with more features. Adjusted R² penalizes for adding features that don't improve the model: . Use adjusted R² when comparing models with different numbers of features.
Q2: Why is linear regression still useful in the age of deep learning? A: Linear regression is fast, interpretable, requires little data, and provides a strong baseline. Coefficients directly show feature importance and direction of effect. Many complex models start by comparing against linear regression.
Q3: When would you use gradient descent over the normal equation? A: When the number of features is large () since the normal equation is . Also when the dataset doesn't fit in memory (stochastic gradient descent). The normal equation is faster for small .
Q4: How do you handle categorical features in linear regression? A: Use one-hot encoding for nominal categories (no order) or ordinal encoding for ordered categories. This creates binary indicator variables that linear regression can use. Be careful with the dummy variable trap (perfect multicollinearity).
Q5: What are the consequences of violating linear regression assumptions? A: (1) Non-linearity → biased predictions, (2) Heteroscedasticity → inefficient standard errors, (3) Non-normal residuals → invalid hypothesis tests, (4) Multicollinearity → unstable coefficients, (5) Autocorrelation → underestimated standard errors.
Q6: Explain the relationship between linear regression and correlation. A: Correlation () measures the strength and direction of the linear relationship between two variables. (from regression) equals for simple linear regression — it measures the proportion of variance explained. Correlation implies association, not causation.
Q7: How would you improve a linear regression model that underperforms? A: (1) Add polynomial or interaction terms for nonlinear relationships, (2) Apply regularization (Ridge/Lasso) if overfitting, (3) Remove irrelevant features, (4) Handle outliers and influential points, (5) Engineer better features, (6) Consider nonlinear models.
Practice Exercise
Challenge: House Price Prediction Pipeline
Build a complete linear regression pipeline from scratch:
import numpy as np
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
# Generate synthetic house price data
np.random.seed(42)
N = 500
sqft = np.random.uniform(800, 4000, N)
bedrooms = np.random.randint(1, 6, N)
age = np.random.uniform(0, 50, N)
# True relationship: price = 150*sqft + 20000*bedrooms - 1000*age + noise
price = 150*sqft + 20000*bedrooms - 1000*age + np.random.normal(0, 30000, N)
X = np.column_stack([sqft, bedrooms, age])
y = price
# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Scale features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
# Train linear regression
model = LinearRegression()
model.fit(X_train_scaled, y_train)
# Predictions
y_pred = model.predict(X_test_scaled)
# Evaluate
print("Coefficients:", dict(zip(['sqft', 'bedrooms', 'age'], model.coef_)))
print(f"Intercept: {model.intercept_:.2f}")
print(f"MSE: {mean_squared_error(y_test, y_pred):.2f}")
print(f"RMSE: {np.sqrt(mean_squared_error(y_test, y_pred)):.2f}")
print(f"MAE: {mean_absolute_error(y_test, y_pred):.2f}")
print(f"R²: {r2_score(y_test, y_pred):.4f}")
# Compare with and without scaling
model_unscaled = LinearRegression().fit(X_train, y_train)
y_pred_unscaled = model_unscaled.predict(X_test)
print(f"\nUnscaled R²: {r2_score(y_test, y_pred_unscaled):.4f}")
print(f"Scaled R²: {r2_score(y_test, y_pred):.4f}")
Bonus challenges:
- Add polynomial features (sqft², sqft×bedrooms) and compare performance
- Implement Ridge regression from scratch using gradient descent
- Plot residual diagnostics (residuals vs predicted, Q-Q plot)
Comparison Table
Regression Algorithms Comparison
| Algorithm | Solution | Complexity | Interpretability | Best For |
|---|---|---|---|---|
| Linear (OLS) | Closed-form: | O(d³) | ★★★★★ | Small-medium d, baseline |
| Linear (GD) | Iterative: | O(Nd × epochs) | ★★★★★ | Large N or d |
| Ridge (L2) | O(d³) | ★★★★☆ | Multicollinearity | |
| Lasso (L1) | Proximal gradient | O(Nd × epochs) | ★★★★☆ | Feature selection |
| Polynomial | OLS on expanded features | O((dp)^3) | ★★★☆☆ | Nonlinear relationships |
Key Formulas Reference
| Formula | Expression | Context |
|---|---|---|
| Simple LR | One feature | |
| Multiple LR | Multiple features | |
| Normal Equation | OLS closed-form | |
| GD Update | Iterative optimization | |
| MSE | Loss function | |
| R² | Variance explained | |
| Adjusted R² | Penalizes extra features |
Key Takeaways
What to Learn Next
Classification with probability — from linear to sigmoid.
Prevent overfitting with Ridge, Lasso, and Elastic Net.
How to know if your model actually works — beyond accuracy.