Ensemble Diversity: Theory and Measurement

Module: Machine Learning | Difficulty: Advanced

Ambiguity Decomposition

Diversity Measures

| Measure | Formula | Range | |---------|---------|-------| | Q-Statistic | | [-1, 1] | | Correlation | | [-1, 1] | | Disagreement | | [0, 1] |

Negative Correlation Learning

Accuracy-Diversity Trade-off

import numpy as np

def ensemble_diversity(predictions, labels):
    n_models = len(predictions)
    diversity = np.zeros((n_models, n_models))
    for i in range(n_models):
        for j in range(i+1, n_models):
            disagree = (predictions[i] != predictions[j]).mean()
            diversity[i,j] = diversity[j,i] = disagree
    return diversity.mean()

def q_statistic(predictions, labels):
    n_models = len(predictions)
    q_stats = []
    for i in range(n_models):
        for j in range(i+1, n_models):
            correct_i = (predictions[i] == labels)
            correct_j = (predictions[j] == labels)
            a = (correct_i & correct_j).sum()
            b = (correct_i & ~correct_j).sum()
            c = (~correct_i & correct_j).sum()
            d = (~correct_i & ~correct_j).sum()
            q = (a*d - b*c) / (a*d + b*c + 1e-10)
            q_stats.append(q)
    return np.mean(q_stats)

Research Insight: Negative correlation learning explicitly encourages diversity by penalizing correlated errors. The key insight is that diversity should be measured in the context of ensemble accuracy, not independently.