Ensemble Methods
Many Trees Make a Forest — The Power of Ensemble Learning
Random Forest builds hundreds of decision trees and merges their predictions to achieve higher accuracy and stability. By combining bagging with random feature selection, it reduces overfitting while maintaining the interpretability of individual trees.
- Bootstrap Aggregating — reduces variance by averaging predictions from multiple trees trained on different data samples
- Random Feature Selection — decorrelates trees by considering only a subset of features at each split
- Out-of-Bag Evaluation — provides a free validation estimate without needing a separate holdout set
"The forest is much wiser than any single tree."
Random Forest — Complete Guide
Random Forest builds many decision trees and combines their predictions. It's one of the most popular and effective ML algorithms.
How Random Forest Works
Bootstrap Sampling Diagram
Random Forest = Bagging + Random Feature Selection
Step 1: Bootstrap Sampling
Create N random samples (with replacement)
Each sample ~63% of original data
~37% left out (out-of-bag samples)
Step 2: Train Decision Tree on Each Sample
At each split, consider only √p features (classification)
Or p/3 features (regression)
This decorrelates the trees
Step 3: Aggregate Predictions
Classification: Majority vote
Regression: Average
Why it works:
Each tree is different (bootstrap + random features)
Errors of individual trees cancel out
Combining reduces variance without increasing bias
Parallel Forest Architecture
Feature Importance Diagram
Mathematical Foundation
Variance Reduction via Bagging
For
independent trees with variance
and pairwise correlation
:
As
, the second term vanishes, leaving:
Key insight: Reducing
(tree correlation) reduces ensemble variance. Random feature selection achieves this by ensuring trees split on different feature subsets.
Optimal Number of Features
For classification with
total features, the theoretical optimum is:
For regression,
typically works well.
Python Implementation
Out-of-Bag (OOB) Evaluation
The OOB error estimator:
where
is the prediction for sample
using only trees that did not include
in their bootstrap sample.
Each tree sees ~63% of data
The remaining ~37% (OOB samples) can be used for evaluation
rf = RandomForestClassifier(oob_score=True)
rf.fit(X, y)
print(f"OOB Score: {rf.oob_score_:.3f}")
Advantage: No need for separate validation set!
Hyperparameters
n_estimators: Number of trees
More = better (up to a point)
100-500 is usually good
Diminishing returns after 500
max_depth: Maximum tree depth
None = unlimited (may overfit)
10-30 is usually good
Deeper = more complex
min_samples_split: Minimum samples to split
2 = default (grow fully)
5-20 = more regularization
Higher = simpler trees
max_features: Features per split
'sqrt' = √p features (classification)
'log2' = log₂(p) features
0.3 = 30% of features
Bias-Variance Analysis
Key Takeaways
What to Learn Next
-> Decision Trees Understand the building blocks of Random Forest — how individual trees split data and make predictions.
-> XGBoost Learn the gradient boosting alternative that often outperforms Random Forest on structured data.
-> Ensemble Methods Explore the broader theory behind bagging, boosting, and stacking ensemble strategies.
-> Model Evaluation Master cross-validation, bias-variance tradeoff, and metrics for assessing Random Forest performance.
-> Interpretability Use SHAP and LIME to explain what your Random Forest model learned from the data.
-> Feature Engineering Create better input features that help Random Forest models achieve even higher accuracy.