Unsupervised Learning

Finding Structure in Unlabeled Data — The Art of Grouping

Clustering algorithms discover natural groups in data without labels. From customer segmentation to image compression, clustering reveals hidden patterns.

K-Means — Partitional clustering with centroids
DBSCAN — Density-based clustering for arbitrary shapes
Hierarchical Clustering — Building dendrograms of relationships

"Clustering is the art of finding groups in data." — Trevor Hastie

Prerequisites

Before diving in, make sure you're comfortable with:

Linear Algebra — Euclidean distance, centroids, vector means
Probability — Distributions, density estimation (for DBSCAN)
Python — NumPy, scikit-learn basics
Feature Scaling — Why normalization matters for distance-based algorithms

Learning Objectives

After completing this tutorial, you will be able to:

Implement K-Means clustering and explain its limitations
Apply DBSCAN for arbitrary-shaped clusters and outlier detection
Perform hierarchical clustering and interpret dendrograms
Evaluate clustering quality using Silhouette, Davies-Bouldin, and Calinski-Harabasz
Choose the right algorithm for your data characteristics
Handle high-dimensional clustering with dimensionality reduction

Clustering — Complete Guide

Clustering is the task of grouping similar data points together without predefined labels.

Types of Clustering

K-Means Clustering

from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.preprocessing import StandardScaler
import numpy as np

# Generate sample data
X, y_true = make_blobs(n_samples=300, centers=4, cluster_std=0.60, random_state=0)

# K-Means clustering
kmeans = KMeans(n_clusters=4, init='k-means++', n_init=10, max_iter=300, random_state=42)
y_kmeans = kmeans.fit_predict(X)

print(f"Cluster centers:\n{kmeans.cluster_centers_}")
print(f"Inertia (WCSS): {kmeans.inertia_:.2f}")
print(f"Iterations to converge: {kmeans.n_iter_}")

Choosing K: Elbow Method & Silhouette

from sklearn.metrics import silhouette_score, silhouette_samples
import matplotlib.pyplot as plt

K_range = range(2, 11)
inertias = []
silhouette_scores = []

for k in K_range:
    km = KMeans(n_clusters=k, random_state=42, n_init=10)
    labels = km.fit_predict(X)
    inertias.append(km.inertia_)
    silhouette_scores.append(silhouette_score(X, labels))

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))
ax1.plot(K_range, inertias, 'bo-', linewidth=2)
ax1.set_xlabel('Number of clusters (K)')
ax1.set_ylabel('Inertia (WCSS)')
ax1.set_title('Elbow Method')

ax2.plot(K_range, silhouette_scores, 'go-', linewidth=2)
ax2.set_xlabel('Number of clusters (K)')
ax2.set_ylabel('Silhouette Score')
ax2.set_title('Silhouette Analysis')

plt.tight_layout()
plt.savefig("choosing_k.png", dpi=150)
plt.show()

K-Means Limitations

DBSCAN

from sklearn.cluster import DBSCAN
from sklearn.datasets import make_moons
import numpy as np

# Non-spherical data
X_moons, y_moons = make_moons(n_samples=300, noise=0.1, random_state=42)

# DBSCAN
dbscan = DBSCAN(eps=0.2, min_samples=5)
labels = dbscan.fit_predict(X_moons)

n_clusters = len(set(labels)) - (1 if -1 in labels else 0)
n_noise = list(labels).count(-1)
print(f"Clusters found: {n_clusters}")
print(f"Noise points: {n_noise}")
print(f"Cluster labels: {set(labels)}")

# Core samples mask
core_mask = np.zeros(len(labels), dtype=bool)
core_mask[dbscan.core_sample_indices_] = True
print(f"Core points: {core_mask.sum()}")

Hierarchical Clustering

from sklearn.cluster import AgglomerativeClustering
from scipy.cluster.hierarchy import dendrogram, linkage
import matplotlib.pyplot as plt

# Agglomerative clustering
agg = AgglomerativeClustering(n_clusters=3, linkage='ward')
labels = agg.fit_predict(X)

# Dendrogram
Z = linkage(X, method='ward')
plt.figure(figsize=(12, 5))
dendrogram(Z, truncate_mode='lastp', p=30, leaf_rotation=90)
plt.title('Hierarchical Clustering Dendrogram')
plt.xlabel('Cluster size')
plt.ylabel('Distance')
plt.savefig("dendrogram.png", dpi=150)
plt.show()

Clustering Evaluation Metrics

from sklearn.metrics import silhouette_score, davies_bouldin_score, calinski_harabasz_score

for k in range(2, 8):
    km = KMeans(n_clusters=k, random_state=42, n_init=10)
    labels = km.fit_predict(X)
    print(f"K={k}: Silhouette={silhouette_score(X, labels):.3f}, "
          f"Davies-Bouldin={davies_bouldin_score(X, labels):.3f}, "
          f"Calinski-Harabasz={calinski_harabasz_score(X, labels):.0f}")

Real-World Applications

Customer Segmentation

Retailers cluster customers by purchase behavior (RFM analysis: Recency, Frequency, Monetary). K-Means segments customers into "loyal," "at-risk," "bargain-hunter" groups for targeted marketing campaigns.

Image Compression (Color Quantization)

K-Means clusters pixel colors into K groups, replacing each pixel with its centroid color. K=16 reduces a 24-bit image to 4-bit while maintaining visual quality — used in GIF compression and mobile image optimization.

Anomaly Detection

DBSCAN identifies outliers (noise points) in network traffic, fraud detection, and manufacturing quality control. Points that don't belong to any dense cluster are flagged for investigation.

Document Clustering

Organizing large document collections into topic groups without labels. TF-IDF vectors + hierarchical clustering creates document taxonomies; DBSCAN finds topic clusters of varying size.

Gene Expression Analysis

Clustering genes by expression patterns reveals functional groups. Hierarchical clustering with dendrograms is standard in bioinformatics for identifying co-regulated genes.

Image Segmentation

Pixel-level clustering separates foreground from background or identifies objects. Mean Shift (a density-based method) is commonly used in medical imaging and autonomous driving.

Common Mistakes & How to Avoid Them

Mistake 1: Not scaling features before clustering

Problem: Features with larger magnitudes dominate distance calculations
Solution: Always use StandardScaler or MinMaxScaler before K-Means or DBSCAN

Mistake 2: Using K-Means on non-spherical clusters

Problem: K-Means assumes spherical, equally-sized clusters — fails on moons, rings, or elongated shapes
Solution: Use DBSCAN or spectral clustering for non-convex shapes

Mistake 3: Choosing K arbitrarily without analysis

Problem: Wrong K gives meaningless clusters
Solution: Use elbow method, silhouette analysis, and domain knowledge to select K

Mistake 4: Ignoring DBSCAN's parameter sensitivity

Problem: Small changes in eps or min_samples dramatically change results
Solution: Use k-distance plot to choose eps; grid search over parameters

Mistake 5: Interpreting clusters as causal

Problem: Clustering finds patterns, not causation — correlation ≠ causation
Solution: Use clusters as exploratory starting points, then validate with domain expertise

Mistake 6: Using accuracy to evaluate clustering

Problem: Clustering is unsupervised — there are no ground truth labels (usually)
Solution: Use internal metrics (silhouette, Davies-Bouldin) or external metrics (if labels exist: ARI, NMI)

Interview Questions

Q1: What is the difference between K-Means and DBSCAN? A: K-Means partitions into K spherical clusters by minimizing WCSS; requires K, sensitive to initialization/outliers. DBSCAN finds arbitrary-shaped clusters based on density; no K needed, handles noise, but struggles with varying densities. K-Means is faster (O(NK) vs O(N log N)).

Q2: How do you choose the number of clusters (K)? A: Methods: (1) Elbow method — plot inertia vs K, look for the "elbow"; (2) Silhouette analysis — maximize average silhouette; (3) Gap statistic — compare to null distribution; (4) Domain knowledge — often the most reliable guide.

Q3: What are the limitations of K-Means? A: Assumes spherical clusters of similar size, sensitive to initialization and outliers, requires K in advance, fails with varying densities or non-convex shapes. Mitigations: K-Means++ initialization, multiple runs (n_init), scaling, or using alternatives like DBSCAN.

Q4: Explain DBSCAN's parameters and how to choose them. A: eps (ε): neighborhood radius — choose using k-distance plot (elbow in sorted distances). min_samples: minimum core points — higher values reduce noise but may merge clusters. Rule of thumb: min_samples ≥ d+1 where d is feature dimension.

Q5: When would you use hierarchical clustering over K-Means? A: When you need a dendrogram to explore cluster relationships at multiple scales, when K is unknown, or when you want deterministic results (no random initialization). Hierarchical is better for small datasets; K-Means scales better.

Q6: What is the difference between agglomerative and divisive hierarchical clustering? A: Agglomerative (bottom-up): starts with N clusters, merges closest pairs iteratively. Divisive (top-down): starts with one cluster, recursively splits. Agglomerative is more common (O(N²) vs O(2^N) for naive divisive).

Q7: Can clustering be used for feature engineering? A: Yes — cluster membership as a categorical feature, distance to cluster centroids as features, or cluster-level statistics (mean, std per cluster) can improve supervised learning models. This is common in Kaggle competitions.

Practice Exercise

Challenge: Compare Clustering Algorithms

import numpy as np
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans, DBSCAN, AgglomerativeClustering
from sklearn.datasets import make_moons, make_circles, make_blobs
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import silhouette_score

# Generate different dataset shapes
datasets = [
    ("Blobs", make_blobs(n_samples=300, centers=4, cluster_std=0.6, random_state=42)),
    ("Moons", make_moons(n_samples=300, noise=0.1, random_state=42)),
    ("Circles", make_circles(n_samples=300, noise=0.1, factor=0.5, random_state=42)),
]

algorithms = [
    ("K-Means", KMeans(n_clusters=4, random_state=42, n_init=10)),
    ("DBSCAN", DBSCAN(eps=0.2, min_samples=5)),
    ("Agglomerative", AgglomerativeClustering(n_clusters=4)),
]

fig, axes = plt.subplots(3, 3, figsize=(12, 12))
for i, (data_name, (X, y)) in enumerate(datasets):
    X_scaled = StandardScaler().fit_transform(X)
    for j, (alg_name, alg) in enumerate(algorithms):
        ax = axes[i, j]
        labels = alg.fit_predict(X_scaled)
        n_clusters = len(set(labels)) - (1 if -1 in labels else 0)
        if n_clusters > 1:
            sil = silhouette_score(X_scaled, labels)
        else:
            sil = 0

        ax.scatter(X_scaled[:, 0], X_scaled[:, 1], c=labels, cmap='viridis', s=30)
        ax.set_title(f"{data_name} + {alg_name}\nK={n_clusters}, Sil={sil:.2f}")

plt.tight_layout()
plt.savefig("clustering_comparison.png", dpi=150)
plt.show()

Bonus challenges:

Implement K-Means from scratch using only NumPy
Find optimal DBSCAN parameters using k-distance plot
Compare different linkage criteria in hierarchical clustering

Key Formulas Reference

Formula	Expression	Context
K-Means Objective		Minimize WCSS
Silhouette		Cluster quality metric
Davies-Bouldin		Lower = better
Calinski-Harabasz		Higher = better

Key Takeaways

What to Learn Next

-> Dimensionality Reduction PCA, t-SNE, and UMAP for visualizing and preprocessing clustered data.

-> Model Evaluation Comprehensive metrics for evaluating unsupervised and supervised models.

-> Deep Learning Clustering Deep Embedded Clustering (DEC) and autoencoder-based approaches.

Clustering — Complete Guide with Visualizations