Supervised Learning
Instance-Based Learning — Your Neighbors Tell the Story
KNN classifies new points by looking at the K closest training examples. It is simple, intuitive, and requires no training phase.
- Lazy Learner — No training phase, all computation at prediction time
- Distance Metrics — Euclidean, Manhattan, and cosine similarity
- Curse of Dimensionality — Why KNN struggles with too many features
"Tell me who your neighbors are, and I'll tell you who you are."
K-Nearest Neighbors — Complete Guide
KNN is the simplest ML algorithm — it classifies a point by looking at its K closest neighbors.
How KNN Works
Distance Metrics
Choosing K
Weighted KNN
Curse of Dimensionality
# Demonstration: distances converge in high dimensions
import numpy as np
for d in [2, 5, 10, 50, 100, 500]:
pts = np.random.rand(100, d)
dists = np.sqrt(((pts[:,None] - pts[None,:])**2).sum(2))
np.fill_diagonal(dists, np.inf)
ratio = dists.max(axis=1).mean() / dists.min(axis=1).mean()
print(f"d={d:3d}: d_max/d_min = {ratio:.2f}")
# Output: d_max/d_min → 1 as d → ∞
Key Takeaways
What to Learn Next
-> Decision Trees If-then rules that learn — the most interpretable algorithm.
-> Clustering Grouping the ungrouped — finding hidden structure in data.
-> Dimensionality Reduction Reduce features while preserving information with PCA and t-SNE.