Information Theory for Machine Learning

Module: Machine Learning | Difficulty: Advanced

Entropy

Mutual Information

KL Divergence

Properties

| Property | Entropy | MI | KL | |----------|---------|----|----| | Non-negative | | | | | Symmetric | Yes | Yes | No | | Additive | | | - |

Feature Selection via MI

import numpy as np
from sklearn.metrics import mutual_info_score

def entropy(x, bins=20):
    hist, _ = np.histogram(x, bins=bins, density=True)
    hist = hist[hist > 0]
    return -np.sum(hist * np.log(hist + 1e-10)) * np.diff(np.linspace(x.min(), x.max(), bins+1))[0]

def mutual_information(x, y, bins=20):
    hist_2d, _, _ = np.histogram2d(x, y, bins=bins)
    p_xy = hist_2d / hist_2d.sum()
    p_x = p_xy.sum(axis=1)
    p_y = p_xy.sum(axis=0)
    mi = 0
    for i in range(bins):
        for j in range(bins):
            if p_xy[i,j] > 0:
                mi += p_xy[i,j] * np.log(p_xy[i,j] / (p_x[i] * p_y[j]))
    return mi

Research Insight: Mutual information captures non-linear dependencies that correlation misses. For feature selection, MI-based methods outperform correlation-based methods when the relationship between features and target is non-linear.