Introduction to Probability
Probability Theory
The Mathematics of Uncertainty
Probability is the mathematical framework for quantifying uncertainty. It assigns a number between 0 and 1 to an event, measuring how likely that event is to occur.
- Classical interpretation — Equal likelihood of outcomes; the fair coin, the rolled die
- Frequentist interpretation — Long-run relative frequency from infinite repetitions
- Subjective interpretation — Personal degree of belief updated by evidence
- Kolmogorov axioms — The three mathematical rules that make probability work
Probability is not about certainty — it is about quantifying what we do not know.
What is Probability?
Definition
Probability is the mathematical framework for quantifying uncertainty. It assigns a number between 0 and 1 to an event, measuring how likely that event is to occur. A probability of 0 means the event is impossible; a probability of 1 means it is certain.
The foundations of modern probability were laid by Kolmogorov (1933), who axiomatized probability as a measure on a set of outcomes.
Three Interpretations of Probability
| Interpretation | Definition | Formalization | Example |
|---|---|---|---|
| Classical | Equal likelihood of outcomes | Fair coin: | |
| Frequentist | Long-run relative frequency | 48 heads in 100 tosses -> | |
| Subjective | Personal degree of belief | Bayesian: encodes belief | "I'm 80% sure it will rain" |
The Axioms of Probability
Immediate Consequences of the Axioms
The Addition Rule
For mutually exclusive events ():
Conditional Probability and Independence
The Multiplication Rule
Total Probability and Bayes' Theorem
Bayes' theorem is the foundation of Bayesian statistics: it updates prior beliefs in light of observed data to produce posterior beliefs .
Counting Principles
Probability in Machine Learning
| ML Application | Probability Usage | Why |
|---|---|---|
| Classification | P(class | features) | Core of supervised learning |
| Naive Bayes | P(feature | class) × P(class) | Text classification baseline |
| Bayesian optimization | P(optimal params | data) | Hyperparameter tuning |
| Uncertainty estimation | Confidence intervals | Trustworthy predictions |
import numpy as np
from sklearn.naive_bayes import GaussianNB
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.3)
# Naive Bayes: applies Bayes' theorem directly
model = GaussianNB()
model.fit(X_train, y_train)
proba = model.predict_proba(X_test[:3])
print("Naive Bayes predictions (probability):")
for i, p in enumerate(proba):
print(f" Sample {i}: {p.round(3)} → class {np.argmax(p)}")
print(f"Accuracy: {model.score(X_test, y_test):.3f}")
print("ML IS applied probability theory!")