1. What Is Anomaly Detection?
Anomaly detection identifies data points that deviate significantly from the majority of the data. These outliers can indicate fraud, system faults, scientific discoveries, or data quality issues.
Types of Anomalous Points
Anomaly Categories
| Type | Definition | Example |
|---|---|---|
| Point | A single observation far from others | Credit card fraud |
| Contextual | Abnormal given its context | 90°F in December |
| Collective | A group of points anomalous together | Coordinated bot attack |
| Global | Outlier vs. entire dataset | Sensor malfunction reading |
| Local | Outlier only within its neighborhood | Normal in global sense but rare locally |
2. Statistical Methods
Z-Score Method
The Z-score measures how many standard deviations a point is from the mean:
A point is flagged as anomalous if (commonly ).
Limitations: Assumes Gaussian distribution; sensitive to extreme values in and (masking effect).
Modified Z-Score (MAD)
Uses median absolute deviation for robustness:
Points with are flagged.
IQR Method
The interquartile range defines fences:
Grubbs' Test
Tests whether the most extreme value is an outlier under normality:
Reject if , where:
3. Isolation Forest
Core Insight
Anomalies are few and different – they are easier to isolate than normal points. Isolation Forest explicitly isolates anomalies by random recursive partitioning.
Algorithm
-
Build an Isolation Tree (iTree):
- Randomly select a feature.
- Randomly select a split value between the feature's min and max.
- Recurse on left and right partitions until isolation or depth limit.
-
Build the forest: Repeat step 1 for trees.
-
Score each point:
where is the average path length of across all trees, and:
is the average path length of unsuccessful search in a BST, with (Euler–Mascheroni constant).
- : highly anomalous
- : normal
Implementation
Key Hyperparameters
| Parameter | Description |
|---|---|
n_estimators | Number of trees (more = more stable) |
max_samples | Subsample size per tree () |
contamination | Expected proportion of anomalies |
max_features | Features per tree (default 1.0) |
4. Local Outlier Factor (LOF)
LOF detects local outliers by comparing the density of a point to its neighbors' densities.
Distance to k-th Nearest Neighbor
Reachability Distance
This smoothing prevents fluctuations from close neighbors.
Local Reachability Density
LOF Score
- : point is in a region of similar density
- : point is in a sparser region than neighbors → anomalous
5. DBSCAN for Anomaly Detection
DBSCAN labels points as core, border, or noise. Noise points are natural anomaly candidates.
Definitions
| Term | Definition |
|---|---|
| ε-neighborhood | |
| Core point | |
| Border point | Not core, but within ε of a core point |
| Noise point | Neither core nor border → anomaly |
Parameter Selection
- eps (ε): Use k-distance graph – sort distances to k-th neighbor and look for the "elbow"
- MinPts: Rule of thumb: (where is feature dimensionality)
6. Autoencoder-Based Detection
Autoencoders learn a compressed representation of normal data. Anomalies produce high reconstruction error because the model has not learned to encode them.
Architecture
Training Objective
Train only on normal data (or assume most data is normal). At inference:
Variants
| Variant | Key Idea |
|---|---|
| Vanilla AE | Standard reconstruction error |
| Variational AE (VAE) | Use |
| Denoising AE | Train to reconstruct from corrupted input |
| LSTM-AE | Temporal autoencoder for time-series anomalies |
Implementation
7. Evaluation Metrics
Anomaly detection is inherently imbalanced – standard accuracy is misleading.
Confusion Matrix for Anomalies
| Predicted Normal | Predicted Anomaly | |
|---|---|---|
| Actual Normal | TN | FP |
| Actual Anomaly | FN | TP |
Key Metrics
Precision and Recall:
F1 Score:
AUROC (Area Under ROC Curve): Threshold-independent metric; ranks anomalous points higher than normal points.
where is a random anomaly and is a random normal point.
AUPRC (Area Under Precision-Recall Curve): More informative than AUROC when anomalies are extremely rare.
Metrics That Don't Require Labels
| Metric | Description |
|---|---|
| Local Outlier Factor | Avg LOF score of flagged points |
| Silhouette Score | Separation of anomaly vs. normal clusters |
| Mass-based | Fraction of total mass assigned to anomalies |
8. Complete Python Implementation
Summary
| Method | Strengths | Weaknesses | Best For |
|---|---|---|---|
| Z-Score / IQR | Simple, interpretable | Gaussian assumption | 1-D, univariate data |
| Isolation Forest | Scalable, no distance computation | Random splits reduce precision | High-dimensional data |
| LOF | Captures local structure | without indexing | Varying-density clusters |
| DBSCAN | No distribution assumption | Sensitive to ε, MinPts | Spatial data, known density |
| Autoencoder | Non-linear, powerful | Needs training data, tuning | Complex high-dim, images, sequences |
Key takeaways:
- No single method dominates – ensemble multiple detectors for robustness
- Feature engineering (domain-specific features) often matters more than algorithm choice
- Threshold selection is critical – use precision-recall tradeoffs aligned with business costs
- For time-series, use temporal models (LSTM-AE, SR) rather than static methods