Bayes' Theorem
Probability Theory
How Evidence Updates What You Believe
Bayes' theorem is the mathematical engine of inductive reasoning. It provides a systematic way to update probabilities in light of new evidence.
- Prior to posterior — Start with a belief, see evidence, get an updated belief
- Base rate neglect — The most common Bayesian error; always account for prevalence
- Medical testing — Positive test result does not mean you have the disease
- Naive Bayes — The algorithm that powers spam filters and text classifiers
Bayes' theorem is the only mathematically valid way to update beliefs with evidence.
What is Bayes' Theorem?
Definition
Bayes' theorem is the mathematical engine of inductive reasoning. It provides a systematic way to update probabilities in light of new evidence.
Derivation
The Bayesian Updating Schema
The key insight: the posterior is a compromise between the prior and the likelihood. When data are abundant, the likelihood dominates and the prior becomes irrelevant. When data are sparse, the prior has more influence.
Sequential Updating
Medical Testing Revisited
Naive Bayes Classification
The Role of the Prior
| Prior Strength | Influence on Posterior | When Appropriate |
|---|---|---|
| Diffuse (uniform) | Data dominate | Little prior knowledge |
| Informative | Prior and data compromise | Strong domain expertise |
| Conjugate | Posterior has same family as prior | Mathematical convenience |
| Jeffreys | Non-informative, transformation-invariant | Objective Bayesian analysis |
Bayes' Theorem in Machine Learning
| ML Application | Bayes' Theorem Usage | Why |
|---|---|---|
| Naive Bayes | P(class|features) ∝ P(features|class)×P(class) | Fast text classification |
| Bayesian optimization | P(optimal|data) | Hyperparameter tuning |
| MAP estimation | θ_MAP = argmax P(data|θ)×P(θ) | Regularization = prior |
| LLMs | P(next_token|context) | GPT, BERT, all transformers |
import numpy as np
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import train_test_split
# Naive Bayes: direct application of Bayes' theorem
texts = ["win money now", "free prize guaranteed", "meeting tomorrow at 3",
"project deadline friday", "claim your reward", "lunch with team"]
labels = [1, 1, 0, 0, 1, 0] # 1=spam, 0=not spam
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)
model = MultinomialNB()
model.fit(X, labels)
# Predict new message
new_msg = ["free money prize"]
new_X = vectorizer.transform(new_msg)
proba = model.predict_proba(new_X)[0]
print(f"Message: '{new_msg[0]}'")
print(f"P(not spam) = {proba[0]:.3f}")
print(f"P(spam) = {proba[1]:.3f}")
print(f"Prediction: {'SPAM' if proba[1] > 0.5 else 'NOT SPAM'}")
print("\nThis IS Bayes' theorem in action!")