Probability Foundations
What is Probability
Sample Space and Events
Axioms of Probability
These axioms form the foundation of all probability theory. The third axiom extends to countably many events via sigma-additivity.
Basic Properties
Derived from the axioms, these properties are essential:
- Complement Rule: , where is the complement of .
- Monotonicity: If , then .
- General Addition Rule: For any events and :
- Bonferroni's Inequality: .
- Boole's Inequality: (union bound).
Counting Principles
Counting techniques are fundamental for computing probabilities in discrete settings.
Extended Counting Formulas
- Permutations with Repetition: for multiset.
- Circular Permutations: for arranging items in a circle.
- Stars and Bars: Number of ways to distribute identical items into distinct bins: .
Inclusion-Exclusion
Example: Two Events
Example: Three Events
Complement Rule
This rule is particularly useful for "at least one" problems:
Python Implementation
import numpy as np
from math import comb, perm, factorial
from itertools import combinations, permutations
# Basic counting
print(f"Permutations P(10,3): {perm(10, 3)}") # 720
print(f"Combinations C(10,3): {comb(10, 3)}") # 120
# Probability of union
def prob_union(p_a, p_b, p_ab):
"""P(A ∪ B) = P(A) + P(B) - P(A ∩ B)"""
return p_a + p_b - p_ab
# Inclusion-Exclusion for three events
def prob_union_three(p_a, p_b, p_c, p_ab, p_ac, p_bc, p_abc):
"""P(A ∪ B ∪ C) using inclusion-exclusion"""
return (p_a + p_b + p_c
- p_ab - p_ac - p_bc
+ p_abc)
# Birthday problem
def birthday_prob(n, days=365):
"""Probability of at least one collision in n birthdays"""
if n > days:
return 1.0
prob_no_collision = 1.0
for i in range(n):
prob_no_collision *= (days - i) / days
return 1 - prob_no_collision
# Simulation approach
def simulate_birthday(n, days=365, trials=100000):
"""Monte Carlo simulation for birthday problem"""
collisions = 0
for _ in range(trials):
birthdays = np.random.randint(0, days, size=n)
if len(np.unique(birthdays)) < n:
collisions += 1
return collisions / trials
# Dice problems
def dice_probability(target_sum, num_dice=2, faces=6):
"""Probability of getting target_sum with num_dice dice"""
# Count favorable outcomes
favorable = 0
total = faces ** num_dice
# Brute force for small cases
if num_dice == 2:
for i in range(1, faces+1):
for j in range(1, faces+1):
if i + j == target_sum:
favorable += 1
else:
# Use dynamic programming
dp = np.zeros((num_dice + 1, target_sum + 1))
dp[0][0] = 1
for die in range(1, num_dice + 1):
for sum_val in range(1, target_sum + 1):
for face in range(1, min(faces, sum_val) + 1):
dp[die][sum_val] += dp[die-1][sum_val-face]
favorable = dp[num_dice][target_sum]
return favorable / total
# Bayes' theorem
def bayes_theorem(p_b_given_a, p_a, p_b):
"""P(A|B) = P(B|A) * P(A) / P(B)"""
return (p_b_given_a * p_a) / p_b
# Example: Medical test
p_disease = 0.01 # Prior probability
p_positive_given_disease = 0.95 # Sensitivity
p_positive_given_healthy = 0.05 # False positive rate
p_healthy = 1 - p_disease
p_positive = (p_positive_given_disease * p_disease +
p_positive_given_healthy * p_healthy)
p_disease_given_positive = bayes_theorem(
p_positive_given_disease, p_disease, p_positive
)
print(f"Probability of disease given positive test: {p_disease_given_positive:.4f}")
# Examples
print(f"\nBirthday problem (23 people): {birthday_prob(23):.4f}")
print(f"Dice: P(sum=7 with 2 dice) = {dice_probability(7):.4f}")
print(f"Simulation: {simulate_birthday(23):.4f}")
Applications in AI/ML
Key Concepts
- Bayes' Theorem:
- Maximum Likelihood Estimation (MLE):
- Posterior Distribution:
Common Mistakes
| Mistake | Explanation | Correct Approach |
|---|---|---|
| Assuming events are independent | only if independent | Check if one event affects the other |
| Confusing permutations and combinations | Order matters for permutations | Use combinations when order doesn't matter |
| Ignoring the complement rule | "At least one" problems are easier with complements | Always consider |
| Double-counting in inclusion-exclusion | Overlapping probabilities counted multiple times | Use the full inclusion-exclusion formula |
| Forgetting conditional probability | in general | Apply Bayes' theorem when conditioning |
| Assuming equally likely outcomes | Not all sample spaces have uniform probabilities | Use the definition of probability, not symmetry |
Interview Questions
1. What is the probability of getting at least one heads in 3 coin flips?
Answer:
2. Explain the difference between independent and mutually exclusive events.
Answer:
- Independent: ; one event doesn't affect the other.
- Mutually exclusive: , so ; they cannot occur together.
- Note: Mutually exclusive events are generally not independent (unless one has probability 0).
3. How would you calculate the probability of being dealt a flush in poker?
Answer: A flush is 5 cards of the same suit. Total 5-card hands: . Favorable: . Probability: .
4. What is Bayes' theorem and why is it important in ML?
Answer: Bayes' theorem: . It updates prior beliefs with evidence. In ML, it's used for classification (Naive Bayes), Bayesian optimization, and probabilistic models.
5. A test has 99% accuracy and 1% prevalence of disease. If you test positive, what's the probability you have the disease?
Answer: Using Bayes' theorem: (50%).
6. How many ways can you arrange 5 books on a shelf?
Answer: permutations.
7. What's the probability of rolling a sum of 7 with two dice?
Answer: Favorable outcomes: (1,6), (2,5), (3,4), (4,3), (5,2), (6,1) = 6. Total: 36. Probability: .
Practice Problems
Probability Inequalities
Quick Reference
| Concept | Formula | Notes |
|---|---|---|
| Sample Space | Set of all outcomes | |
| Event | Subset of outcomes | |
| Complement | Probability event doesn't occur | |
| Addition Rule | For any events | |
| Multiplication Rule | Chain rule | |
| Independence | No influence | |
| Mutually Exclusive | Cannot co-occur | |
| Bayes' Theorem | Update beliefs | |
| Permutations | Order matters | |
| Combinations | Order doesn't matter | |
| Inclusion-Exclusion | Avoid double-counting | |
| Law of Total Probability | Partition of sample space |
Cross-References
- Conditional Probability: See Conditional Probability for deeper dive into .
- Random Variables: See Random Variables for discrete and continuous distributions.
- Distributions: See Common Distributions for Bernoulli, Binomial, Poisson, etc.
- Statistical Inference: See Hypothesis Testing for applying probability in decision-making.
- Machine Learning: See Bayesian Methods for probabilistic ML models.