Advanced Ensemble Methods

What Is Advanced Ensemble Methods?

Advanced Ensemble Methods is a key concept in Machine Learning. Understanding it is essential for building effective data science solutions.

Core Idea: Advanced Ensemble Methods provides a systematic approach to ensemble methods problems by learning patterns from data and applying them to make predictions or decisions.

Key Concepts

Ensemble Learning Fundamentals

Bias-Variance decomposition:

$\text{MSE} = \text{Bias}^2 + \text{Variance} + \text{Noise}$

Bagging (Random Forest) reduces variance:

$\hat{f}_{bag}(x) = \frac{1}{B}\sum_{b=1}^B \hat{f}_b(x)$

Boosting (XGBoost/GBM) reduces bias — sequentially fits residuals:

$F_m(x) = F_{m-1}(x) + \eta h_m(x)$

Where $h_m$ is a weak learner fit to the negative gradient (pseudo-residuals).

Python Implementation

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import load_breast_cancer, load_iris
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, classification_report
import warnings
warnings.filterwarnings("ignore")

# Load example dataset
data = load_breast_cancer()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target

# Prepare data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s  = scaler.transform(X_test)

print(f"Dataset: {X.shape[0]} samples, {X.shape[1]} features")
print(f"Class distribution: {dict(pd.Series(y).value_counts())}")
print(f"Train / Test split: {len(X_train)} / {len(X_test)}")

from sklearn.ensemble import GradientBoostingClassifier, VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier

# Gradient Boosting
model = GradientBoostingClassifier(
    n_estimators=200, learning_rate=0.1,
    max_depth=3, subsample=0.8,
    random_state=42
)
model.fit(X_train_s, y_train)

# Voting ensemble
voting = VotingClassifier(estimators=[
    ("lr",  LogisticRegression(max_iter=1000)),
    ("rf",  RandomForestClassifier(n_estimators=100, random_state=42)),
    ("gb",  GradientBoostingClassifier(n_estimators=100, random_state=42)),
], voting="soft")
voting.fit(X_train_s, y_train)
print(f"Voting accuracy: {voting.score(X_test_s, y_test):.4f}")

Evaluation & Results

# Evaluate model performance
y_pred = model.predict(X_test_s)

print(f"Accuracy  : {accuracy_score(y_test, y_pred):.4f}")
print(f"\nClassification Report:")
print(classification_report(y_test, y_pred,
      target_names=data.target_names))

# Cross-validation for robust estimate
cv_scores = cross_val_score(model, X_train_s, y_train, cv=5, scoring="f1")
print(f"\n5-Fold CV F1: {cv_scores.mean():.4f} ± {cv_scores.std():.4f}")

# Visualise results
fig, axes = plt.subplots(1, 2, figsize=(12, 4))

# Confusion matrix
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
cm = confusion_matrix(y_test, y_pred)
ConfusionMatrixDisplay(cm, display_labels=data.target_names).plot(ax=axes[0])
axes[0].set_title("Confusion Matrix")

# CV scores
axes[1].bar(range(1, 6), cv_scores, color="#3b82f6", edgecolor="white")
axes[1].axhline(cv_scores.mean(), color="red", linestyle="--",
                label=f"Mean={cv_scores.mean():.4f}")
axes[1].set_xlabel("Fold"); axes[1].set_ylabel("F1 Score")
axes[1].set_title("Cross-Validation Scores")
axes[1].legend(); axes[1].grid(True, alpha=0.3, axis="y")

plt.tight_layout()
plt.show()

Comparison with Related Methods

Hyperparameter Tuning

from sklearn.model_selection import GridSearchCV

param_grid = {
    "C":       [0.01, 0.1, 1.0, 10.0],
    "gamma":   ["scale", "auto"],
    "kernel":  ["rbf", "linear"],
}

grid = GridSearchCV(model, param_grid, cv=5,
                    scoring="f1", n_jobs=-1, verbose=0)
grid.fit(X_train_s, y_train)

print(f"Best params : {grid.best_params_}")
print(f"Best CV F1  : {grid.best_score_:.4f}")
print(f"Test F1     : {grid.score(X_test_s, y_test):.4f}")

Key Takeaways

1. Advanced Ensemble Methods is a powerful method for ensemble methods tasks
2. Always scale features before applying distance-based or regularised methods
3. Use cross-validation — never evaluate on the same data used for training
4. Start simple — a strong baseline prevents over-engineering
5. Visualise everything — confusion matrices, learning curves, feature importances