Unsupervised Learning
Curse of Dimensionality — When More Features Hurt, Not Help
Dimensionality reduction compresses high-dimensional data into fewer dimensions while preserving the most important structure and variance.
- PCA — finds orthogonal axes of maximum variance for fast, linear dimensionality reduction
- t-SNE — preserves local neighborhoods for intuitive 2D and 3D visualization
- UMAP — faster than t-SNE with better global structure preservation
"Not everything that can be counted counts, and not everything that counts can be counted."
Dimensionality Reduction — Complete Guide
Dimensionality reduction compresses high-dimensional data into fewer dimensions while preserving important information.
Why Reduce Dimensions?
Curse of Dimensionality Visualization
Curse of Dimensionality:
More dimensions = more data needed
Distances become meaningless
Models overfit
Training becomes slow
Benefits:
Faster training
Less overfitting
Better visualization (2D/3D)
Removes noise
Fewer features = simpler model
PCA (Principal Component Analysis)
PCA Projection Diagram
PCA finds the directions of MAXIMUM VARIANCE:
1. Standardize data
2. Compute covariance matrix
3. Find eigenvectors (principal components)
4. Project data onto top K eigenvectors
PC1: Direction of most variance
PC2: Direction of second most variance (orthogonal to PC1)
...
Explained variance ratio tells you how much info each PC captures:
PC1: 72%
PC2: 15%
PC3: 8%
PC4: 5% -> can probably drop this one
PCA Mathematics
The covariance matrix:
Eigendecomposition:
Explained variance ratio:
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
X_2d = pca.fit_transform(X)
print(f"Explained variance: {pca.explained_variance_ratio_}")
# [0.72, 0.15] — first 2 components explain 87% of variance
t-SNE
t-SNE Visualization
t-SNE preserves LOCAL structure (neighborhoods):
Best for: Visualization (2D/3D)
Not for: Feature reduction for training
How it works:
1. Compute similarities in high-D (Gaussian)
2. Compute similarities in low-D (Student-t)
3. Minimize KL divergence between them
Key parameters:
perplexity: Number of neighbors (5-50)
learning_rate: Step size (10-1000)
n_iter: Number of iterations (1000+)
UMAP
UMAP vs t-SNE Comparison
UMAP = faster, better version of t-SNE:
Advantages over t-SNE:
10x faster
Better preserves global structure
Can transform new data
Better for clustering
preserves both local and global structure
import umap
reducer = umap.UMAP(n_components=2, n_neighbors=15)
X_2d = reducer.fit_transform(X)
Comparison
| Method | Speed | Local | Global | Transform |
|---|---|---|---|---|
| PCA | Fast | No | Yes | Yes |
| t-SNE | Slow | Yes | No | No |
| UMAP | Medium | Yes | Yes | Yes |
| LDA | Fast | No | No | Yes |
Key Takeaways
What to Learn Next
-> Autoencoders Learn the neural network approach to nonlinear dimensionality reduction and representation learning.
-> Clustering Group similar data points using K-Means, DBSCAN, and hierarchical methods.
-> Feature Engineering Create and transform features to improve model performance before dimensionality reduction.
-> Model Evaluation Evaluate whether dimensionality reduction improved or hurt your model's predictive power.
-> Neural Networks Understand the deep learning foundations that autoencoders are built upon.
-> CNNs Apply convolutional architectures to image data where spatial dimensionality matters.