Naive Bayes: Conditional Independence and Extensions

Module: Machine Learning | Difficulty: Advanced

Bayes Theorem

Naive Bayes Assumption

Gaussian Naive Bayes

Laplace Smoothing

Log-Posterior

import numpy as np

class NaiveBayes:
    def __init__(self, alpha=1.0):
        self.alpha = alpha
    def fit(self, X, y):
        self.classes = np.unique(y)
        self.priors = {}
        self.means = {}
        self.vars = {}
        for c in self.classes:
            X_c = X[y == c]
            self.priors[c] = len(X_c) / len(X)
            self.means[c] = X_c.mean(axis=0)
            self.vars[c] = X_c.var(axis=0) + 1e-9
    def _log_likelihood(self, x, c):
        return -0.5 * np.sum(np.log(2*np.pi*self.vars[c]) + (x-self.means[c])**2/self.vars[c])
    def predict(self, X):
        return np.array([max(self.classes, key=lambda c: np.log(self.priors[c]) + self._log_likelihood(x, c)) for x in X])

Research Insight: Despite the strong independence assumption, Naive Bayes often performs well because: (1) classification only needs the correct ranking of posterior probabilities, not exact values, and (2) dependencies between features partially cancel out.