Dimensionality Reduction

What Is Dimensionality Reduction?

Dimensionality Reduction is a key concept in Machine Learning. Understanding it is essential for building effective data science solutions.

Core Idea: Dimensionality Reduction provides a systematic approach to dimensionality reduction problems by learning patterns from data and applying them to make predictions or decisions.

Key Concepts

Mathematical Foundation

PCA finds directions of maximum variance in data.

Given centred data matrix $\mathbf{X}$, compute covariance:

$\Sigma = \frac{1}{n-1}\mathbf{X}^\top\mathbf{X}$

Eigendecomposition: $\Sigma = \mathbf{V}\Lambda\mathbf{V}^\top$

Principal components = eigenvectors $\mathbf{v}_1, \mathbf{v}_2, \ldots$ sorted by eigenvalue.

Explained variance ratio:

$\text{EVR}_k = \frac{\lambda_k}{\sum_{j=1}^p \lambda_j}$

SVD connection: $\mathbf{X} = \mathbf{U}\Sigma\mathbf{V}^\top$ — columns of $\mathbf{V}$ are principal components.

Python Implementation

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import load_breast_cancer, load_iris
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, classification_report
import warnings
warnings.filterwarnings("ignore")

# Load example dataset
data = load_breast_cancer()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target

# Prepare data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s  = scaler.transform(X_test)

print(f"Dataset: {X.shape[0]} samples, {X.shape[1]} features")
print(f"Class distribution: {dict(pd.Series(y).value_counts())}")
print(f"Train / Test split: {len(X_train)} / {len(X_test)}")

from sklearn.decomposition import PCA
import matplotlib.pyplot as plt

pca = PCA(n_components=0.95, random_state=42)   # keep 95% variance
X_pca = pca.fit_transform(X_train_s)

print(f"Original features  : {X_train_s.shape[1]}")
print(f"PCA components     : {pca.n_components_}")
print(f"Variance explained : {pca.explained_variance_ratio_.sum():.4f}")

# Scree plot
plt.figure(figsize=(9, 4))
plt.plot(range(1, len(pca.explained_variance_ratio_)+1),
         pca.explained_variance_ratio_.cumsum(), "bo-")
plt.axhline(0.95, color="red", linestyle="--", label="95% threshold")
plt.xlabel("Components"); plt.ylabel("Cumulative Variance")
plt.title("PCA Scree Plot"); plt.legend(); plt.grid(True, alpha=0.3)
plt.show()

model = None  # placeholder — use PCA components downstream

Evaluation & Results

# Evaluate model performance
y_pred = model.predict(X_test_s)

print(f"Accuracy  : {accuracy_score(y_test, y_pred):.4f}")
print(f"\nClassification Report:")
print(classification_report(y_test, y_pred,
      target_names=data.target_names))

# Cross-validation for robust estimate
cv_scores = cross_val_score(model, X_train_s, y_train, cv=5, scoring="f1")
print(f"\n5-Fold CV F1: {cv_scores.mean():.4f} ± {cv_scores.std():.4f}")

# Visualise results
fig, axes = plt.subplots(1, 2, figsize=(12, 4))

# Confusion matrix
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
cm = confusion_matrix(y_test, y_pred)
ConfusionMatrixDisplay(cm, display_labels=data.target_names).plot(ax=axes[0])
axes[0].set_title("Confusion Matrix")

# CV scores
axes[1].bar(range(1, 6), cv_scores, color="#3b82f6", edgecolor="white")
axes[1].axhline(cv_scores.mean(), color="red", linestyle="--",
                label=f"Mean={cv_scores.mean():.4f}")
axes[1].set_xlabel("Fold"); axes[1].set_ylabel("F1 Score")
axes[1].set_title("Cross-Validation Scores")
axes[1].legend(); axes[1].grid(True, alpha=0.3, axis="y")

plt.tight_layout()
plt.show()

Comparison with Related Methods

Hyperparameter Tuning

from sklearn.model_selection import GridSearchCV

param_grid = {
    "C":       [0.01, 0.1, 1.0, 10.0],
    "gamma":   ["scale", "auto"],
    "kernel":  ["rbf", "linear"],
}

grid = GridSearchCV(model, param_grid, cv=5,
                    scoring="f1", n_jobs=-1, verbose=0)
grid.fit(X_train_s, y_train)

print(f"Best params : {grid.best_params_}")
print(f"Best CV F1  : {grid.best_score_:.4f}")
print(f"Test F1     : {grid.score(X_test_s, y_test):.4f}")

Key Takeaways

1. Dimensionality Reduction is a powerful method for dimensionality reduction tasks
2. Always scale features before applying distance-based or regularised methods
3. Use cross-validation — never evaluate on the same data used for training
4. Start simple — a strong baseline prevents over-engineering
5. Visualise everything — confusion matrices, learning curves, feature importances