Ensemble Methods: Bagging, Boosting, and Stacking

Module: Machine Learning | Difficulty: Advanced

Bagging

Variance reduction:

AdaBoost

Theorem (Schapire et al., 1998): Training error decreases exponentially:

Gradient Boosting

where fits the negative gradient

Ensemble Diversity

import numpy as np

class GradientBoosting:
    def __init__(self, n_estimators=100, learning_rate=0.1, max_depth=3):
        self.n_estimators = n_estimators
        self.lr = learning_rate
        self.max_depth = max_depth
        self.trees = []
        self.initial = None
    def fit(self, X, y):
        self.initial = y.mean()
        f = np.full(len(y), self.initial)
        for _ in range(self.n_estimators):
            residual = y - f
            tree = DecisionTreeRegressor(max_depth=self.max_depth)
            tree.fit(X, residual)
            f += self.lr * tree.predict(X)
            self.trees.append(tree)
    def predict(self, X):
        f = np.full(X.shape[0], self.initial)
        for tree in self.trees:
            f += self.lr * tree.predict(X)
        return f

Research Insight: Bagging reduces variance by averaging correlated estimators. The key insight is that correlation between base learners determines the variance reduction: lower correlation = greater reduction.