Linear Regression: Math, Code and Assumptions

The Foundation of Machine Learning

Linear regression is the most fundamental algorithm in ML. Despite its simplicity, understanding it deeply provides insight into all supervised learning methods.

1. Simple Linear Regression

Mathematical Formulation

How this diagram works: This diagram shows the core concept of simple linear regression — fitting a straight line through scattered data points to model the relationship between a feature (x) and a target (y). The blue data points represent actual observations, while the purple regression line represents the model's predictions. The red dashed lines (residuals) show the vertical distance between each data point and the line, representing prediction errors. The goal of linear regression is to minimize these residuals by finding the optimal intercept (β₀) and slope (β₁) that produce the smallest total squared error.

2. Cost Function (Ordinary Least Squares)

The cost function is convex — gradient descent finds the global minimum

3. Gradient Descent

4. Multiple Linear Regression

Multiple Regression: Multiple Features → Single Output

        {f.label.split('\n')[0]}


        {f.label.split('\n')[1]}


        β{['₁', '₂', '₃', '₃'][i]}

Linear
Model
ŷ = β₀ + Σβ⊥x⊥
Output
ŷ (Price)

5. Model Evaluation Metrics

R² = 1 - (30/100) = 0.70 (70% variance explained)

6. Assumptions of Linear Regression

{`5 Key Assumptions to Validate

Linearity          y = f(x) is linear
Independence       Errors are independent
Homoscedasticity   Constant variance
Normality          ε ~ N(0, σ²)
No Multicollinearity  Features not correlated`}

Checking Assumptions with Residual Plots

7. Implementation in Python

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score

# Generate sample data
np.random.seed(42)
X = 2 * np.random.rand(100, 1)
y = 4 + 3 * X + np.random.randn(100, 1)

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Fit model
model = LinearRegression()
model.fit(X_train, y_train)

# Predictions
y_pred = model.predict(X_test)

# Evaluate
print(f"Intercept (β₀): {model.intercept_[0]:.4f}")
print(f"Slope (β₁): {model.coef_[0][0]:.4f}")
print(f"R² Score: {r2_score(y_test, y_pred):.4f}")
print(f"RMSE: {np.sqrt(mean_squared_error(y_test, y_pred)):.4f}")

# Visualize
plt.scatter(X_test, y_test, color='blue', alpha=0.6, label='Actual')
plt.plot(X_test, y_pred, color='red', linewidth=2, label='Predicted')
plt.xlabel('Feature')
plt.ylabel('Target')
plt.title('Linear Regression Fit')
plt.legend()
plt.show()

Key Takeaways

1. Linear regression finds the best-fit line through data points
2. Cost function (MSE) measures prediction error — minimize it
3. Gradient descent iteratively updates weights to find minimum
4. R² score tells you how much variance the model explains
5. Validate assumptions before trusting the model
6. Regularization (Ridge/Lasso) prevents overfitting