Bayes' Theorem

Why It Matters

Bayes' Theorem

Prior, Likelihood, Posterior

Bayesian Updating

The Bayesian updating process is iterative — each observation refines the posterior, and the posterior becomes the prior for the next update.

Medical Diagnosis

Medical diagnosis is the canonical application of Bayes' Theorem. It highlights why base rates (prevalence) matter enormously.

Spam Classification

Prior Distributions

The choice of prior significantly impacts inference, especially with limited data. Conjugate priors offer mathematical tractability.

MAP vs MLE

Python Implementation

import numpy as np
from scipy import stats


class NaiveBayesClassifier:
    """Gaussian Naive Bayes classifier for continuous features."""

    def fit(self, X, y):
        self.classes = np.unique(y)
        self.params = {}
        for c in self.classes:
            X_c = X[y == c]
            self.params[c] = {
                "mean": X_c.mean(axis=0),
                "var": X_c.var(axis=0) + 1e-9,  # smoothing
                "prior": len(X_c) / len(X),
            }
        return self

    def _log_likelihood(self, x, mean, var):
        return -0.5 * np.sum(np.log(2 * np.pi * var) + ((x - mean) ** 2) / var)

    def predict(self, X):
        predictions = []
        for x in X:
            posteriors = []
            for c in self.classes:
                log_prior = np.log(self.params[c]["prior"])
                log_lik = self._log_likelihood(
                    x, self.params[c]["mean"], self.params[c]["var"]
                )
                posteriors.append(log_prior + log_lik)
            predictions.append(self.classes[np.argmax(posteriors)])
        return np.array(predictions)


# Bayesian updating for a coin flip (Beta-Binomial)
def bayesian_coin_update(prior_alpha, prior_beta, flips):
    """Sequential Bayesian updating for a biased coin."""
    alpha, beta = prior_alpha, prior_beta
    history = [(alpha, beta)]

    for flip in flips:
        if flip == 1:
            alpha += 1
        else:
            beta += 1
        history.append((alpha, beta))

    return history


# Example usage
prior_alpha, prior_beta = 2, 2  # Beta(2,2) prior — mildly informative
flips = [1, 1, 0, 1, 1, 1, 0, 1, 1, 1]  # observed sequence
history = bayesian_coin_update(prior_alpha, prior_beta, flips)

print("Step | Alpha | Beta | Mean | 95% CI")
print("-" * 50)
for i, (a, b) in enumerate(history):
    mean = a / (a + b)
    ci_low = stats.beta.ppf(0.025, a, b)
    ci_high = stats.beta.ppf(0.975, a, b)
    label = "Prior" if i == 0 else f"Obs {i}"
    print(f"{label:5s} | {a:5.1f} | {b:4.1f} | {mean:.3f} | [{ci_low:.3f}, {ci_high:.3f}]")

Output:

Step | Alpha | Beta  | Mean  | 95% CI
--------------------------------------------------
Prior |   2.0 |   2.0 | 0.500 | [0.079, 0.921]
Obs 1 |   3.0 |   2.0 | 0.600 | [0.170, 0.934]
Obs 2 |   4.0 |   2.0 | 0.667 | [0.263, 0.942]
Obs 3 |   4.0 |   3.0 | 0.571 | [0.245, 0.855]
Obs 4 |   5.0 |   3.0 | 0.625 | [0.306, 0.882]
Obs 5 |   6.0 |   3.0 | 0.667 | [0.359, 0.900]
Obs 6 |   7.0 |   3.0 | 0.700 | [0.400, 0.915]
Obs 7 |   7.0 |   4.0 | 0.636 | [0.363, 0.860]
Obs 8 |   8.0 |   4.0 | 0.667 | [0.410, 0.872]
Obs 9 |   9.0 |   4.0 | 0.692 | [0.444, 0.887]
Obs10 |  10.0 |   4.0 | 0.714 | [0.472, 0.898]

The posterior mean starts at 0.5 (prior belief) and converges toward the observed proportion (7/10 = 0.7) as data accumulates.

Applications in AI/ML

Common Mistakes

Interview Questions

Q1: Why is Bayes' Theorem important in machine learning?

A: Bayes' Theorem provides a principled framework for updating beliefs given evidence. In ML, it enables: (1) incorporating prior knowledge into models, (2) quantifying uncertainty in predictions, (3) building models that improve with data sequentially, and (4) regularizing estimates through priors. It underpins Bayesian neural networks, Gaussian processes, and probabilistic graphical models.

Q2: What is the difference between MLE and MAP?

A: MLE maximizes (the likelihood); MAP maximizes (the posterior). MAP is equivalent to MLE with a log-prior penalty term. For example, with a Gaussian prior, MAP is equivalent to -regularized MLE. MAP produces a point estimate; full Bayesian inference computes the entire posterior distribution.

Q3: A test for a disease with 1% prevalence has 90% sensitivity and 95% specificity. If someone tests positive, what is the probability they have the disease?

A: . Despite the test being quite accurate, only ~15.4% of positive results are true positives due to the low base rate.

Q4: Why does Naive Bayes work well in practice despite its independence assumption?

A: Three reasons: (1) Classification only requires ranking posteriors, not estimating them accurately — errors in can cancel out if the ranking is preserved. (2) With enough training data, the likelihood term dominates and corrects for prior misspecification. (3) Feature dependencies tend to partially cancel each other's effects. Naive Bayes is particularly effective for high-dimensional sparse data like text.

Q5: What is a conjugate prior and why is it useful?

A: A conjugate prior is one where the posterior belongs to the same distribution family as the prior. For example, the Beta distribution is conjugate to the Bernoulli likelihood — after observing data, the posterior is also Beta. This makes computation analytically tractable (no numerical integration needed), enables elegant closed-form updates, and allows intuitive interpretation of hyperparameters as "pseudo-counts."

Q6: How do you choose a prior in practice?

A: Common approaches: (1) Use domain knowledge if available (e.g., known disease prevalence). (2) Use non-informative priors (Jeffreys prior, uniform) when knowledge is limited. (3) Use weakly informative priors (e.g., Beta(2,2)) to regularize without dominating. (4) Always perform prior sensitivity analysis — check if conclusions are robust to different priors. With sufficient data, the likelihood dominates and the prior's influence diminishes.

Practice Problems

Problem 1: Weather Prediction

You observe that on 8 out of 10 days when it was cloudy in the morning, it rained by afternoon. On 3 out of 10 days when it was sunny in the morning, it rained by afternoon. Historically, it rains on 40% of days.

Question: If it is cloudy this morning, what is the probability of rain this afternoon?

Problem 2: Bayesian Spam Filter

A spam filter uses two features: whether the email contains "free" and whether it contains "winner". Training data:

An email contains both "free" and "winner". Should it be classified as spam?

Problem 3: Beta-Binomial Updating

You believe a coin is fair, modeled as a Beta(5, 5) prior. You flip it 20 times and get 14 heads.

Question: What is the posterior distribution? What is the posterior probability that ?

Problem 4: Medical Test with Two Tests

A disease has 2% prevalence. Test A has sensitivity 95% and specificity 90%. Test B (confirmatory) has sensitivity 99% and specificity 98%. If someone tests positive on both tests, what is ?

Quick Reference

Cross-References

• 034 - Probability Conditional — Conditional probability and the Law of Total Probability, prerequisites for Bayes' Theorem.
• 035 - Probability Random Variables — Random variables and distributions underlying Bayesian inference.
• 037 - Statistics Hypothesis Testing — Frequentist hypothesis testing; contrast with Bayesian hypothesis testing.
• 038 - Statistics Regression — Linear regression; Bayesian linear regression uses priors on coefficients.
• 040 - ML Classification — Naive Bayes classifier in the context of broader classification methods.
• 042 - ML Model Evaluation — Metrics for evaluating classifiers; calibration of probabilistic predictions.
• 044 - ML Optimization — MAP estimation and regularization as optimization problems.