Ensemble Methods
Gradient Boosting to the Extreme — Kaggle's Favorite Algorithm
XGBoost builds trees sequentially, with each new tree correcting the errors of the previous ensemble. It dominates Kaggle competitions and remains the gold standard for tabular data.
-
Sequential Learning — each tree learns from the mistakes of all previous trees, steadily reducing bias
-
Regularization — built-in L1 and L2 penalties prevent overfitting on complex datasets
-
Scalability — optimized for speed with parallel tree construction and cache-aware access
"Gradient boosting turns weak learners into strong predictors."
XGBoost and Gradient Boosting — Complete Guide
Gradient Boosting builds trees sequentially — each new tree corrects the errors of previous ones. XGBoost is the most popular implementation.
Boosting vs Bagging
Boosting Sequential Process Diagram
Bagging (Random Forest):
Trees trained INDEPENDENTLY (parallel)
Each tree on different data sample
Reduce variance
Combine by averaging
Boosting (XGBoost):
Trees trained SEQUENTIALLY
Each tree corrects previous errors
Reduce bias AND variance
Combine by weighted sum
Tree Splitting with Gain Diagram
How Gradient Boosting Works
Gradient Boosting Objective Function
where:
is the loss for sample
at step
is the new tree added at step
is the regularization term
Second-Order Taylor Expansion
XGBoost uses a second-order approximation:
where:
(first gradient)
(second gradient)
XGBoost Implementation
Key Hyperparameters
n_estimators: Number of trees (100-1000)
learning_rate: Step size (0.01-0.3)
Lower = need more trees
Higher = need fewer trees
Usually 0.01-0.1
max_depth: Tree depth (3-10)
Shallower = simpler, less overfitting
Deeper = more complex, may overfit
subsample: Row sampling (0.5-1.0)
colsample_bytree: Column sampling (0.5-1.0)
Similar to Random Forest feature sampling
Adds randomness, reduces overfitting
min_child_weight: Minimum child weight (1-10)
gamma: Minimum loss reduction for split (0-5)
XGBoost vs Random Forest
| Feature | Random Forest | XGBoost |
|---------|---------------|---------|
| Training | Parallel | Sequential |
| Speed | Faster | Slower |
| Overfitting | Less likely | More likely |
| Performance | Good | Excellent |
| Hyperparameters | Fewer | More |
| Interpretability | Feature importance | Feature importance |
Key Takeaways
What to Learn Next
Compare the parallel bagging approach to XGBoost's sequential boosting strategy.
Learn the full theory behind bagging, boosting, and stacking ensemble techniques.
Understand the foundational algorithm that XGBoost builds upon and extends.
Master cross-validation and early stopping to find the optimal number of boosting rounds.
Understand L1 and L2 penalties that XGBoost uses to prevent overfitting.
Craft better features to give XGBoost a stronger signal to learn from.