Information Bottleneck: Compression and Prediction

Module: Machine Learning | Difficulty: Advanced

Information Bottleneck Principle

Rate-Distortion

Deep Information Bottleneck

Compression-Prediction Trade-off

| | | | Interpretation | |---------|---------|---------|----------------| | Low | High | High | Overfitting | | Medium | Medium | Medium | Balanced | | High | Low | Low | Underfitting |

import numpy as np

class InformationBottleneck:
    def __init__(self, beta=1.0):
        self.beta = beta
    def mutual_information(self, X, Z, Y):
        # Estimate I(X;Z)
        mi_xz = self._estimate_mi(X, Z)
        # Estimate I(Z;Y)
        mi_zy = self._estimate_mi(Z, Y)
        return mi_xz - self.beta * mi_zy
    def _estimate_mi(self, X, Z, k=5):
        from scipy.spatial.distance import cdist
        n = len(X)
        dists_x = cdist(X, X)
        dists_z = cdist(Z, Z)
        knn_x = np.sort(dists_x, axis=1)[:, k]
        knn_z = np.sort(dists_z, axis=1)[:, k]
        mi = np.mean(np.log(knn_z + 1e-10) - np.log(knn_x + 1e-10))
        return mi

Research Insight: The information bottleneck theory suggests that deep learning optimizes a trade-off between compression and prediction. The fitting phase (reducing ) is followed by a compression phase (reducing ), explaining the double descent phenomenon.