Probability Foundations

Probability Foundations

---

What is Probability

---

Sample Space and Events

---

Axioms of Probability

These axioms form the foundation of all probability theory. The third axiom extends to countably many events via sigma-additivity.

---

Basic Properties

Derived from the axioms, these properties are essential:

• Complement Rule: $P(A^c) = 1 - P(A)$, where $A^c$ is the complement of $A$.
• Monotonicity: If $A \subseteq B$, then $P(A) \leq P(B)$.
• General Addition Rule: For any events $A$ and $B$:

• Bonferroni's Inequality: $P(A \cap B) \geq P(A) + P(B) - 1$.
• Boole's Inequality: $P(\bigcup_{i=1}^n A_i) \leq \sum_{i=1}^n P(A_i)$ (union bound).

---

Counting Principles

Counting techniques are fundamental for computing probabilities in discrete settings.

Extended Counting Formulas

• Permutations with Repetition: $\frac{n!}{n_1! n_2! \cdots n_k!}$ for multiset.
• Circular Permutations: $(n-1)!$ for arranging $n$ items in a circle.
• Stars and Bars: Number of ways to distribute $n$ identical items into $k$ distinct bins: $\binom{n+k-1}{k-1}$.

---

Inclusion-Exclusion

Example: Two Events

Example: Three Events

---

Complement Rule

This rule is particularly useful for "at least one" problems:

---

Python Implementation

import numpy as np
from math import comb, perm, factorial
from itertools import combinations, permutations

# Basic counting
print(f"Permutations P(10,3): {perm(10, 3)}")  # 720
print(f"Combinations C(10,3): {comb(10, 3)}")  # 120

# Probability of union
def prob_union(p_a, p_b, p_ab):
    """P(A ∪ B) = P(A) + P(B) - P(A ∩ B)"""
    return p_a + p_b - p_ab

# Inclusion-Exclusion for three events
def prob_union_three(p_a, p_b, p_c, p_ab, p_ac, p_bc, p_abc):
    """P(A ∪ B ∪ C) using inclusion-exclusion"""
    return (p_a + p_b + p_c 
            - p_ab - p_ac - p_bc 
            + p_abc)

# Birthday problem
def birthday_prob(n, days=365):
    """Probability of at least one collision in n birthdays"""
    if n > days:
        return 1.0
    prob_no_collision = 1.0
    for i in range(n):
        prob_no_collision *= (days - i) / days
    return 1 - prob_no_collision

# Simulation approach
def simulate_birthday(n, days=365, trials=100000):
    """Monte Carlo simulation for birthday problem"""
    collisions = 0
    for _ in range(trials):
        birthdays = np.random.randint(0, days, size=n)
        if len(np.unique(birthdays)) < n:
            collisions += 1
    return collisions / trials

# Dice problems
def dice_probability(target_sum, num_dice=2, faces=6):
    """Probability of getting target_sum with num_dice dice"""
    # Count favorable outcomes
    favorable = 0
    total = faces ** num_dice
    
    # Brute force for small cases
    if num_dice == 2:
        for i in range(1, faces+1):
            for j in range(1, faces+1):
                if i + j == target_sum:
                    favorable += 1
    else:
        # Use dynamic programming
        dp = np.zeros((num_dice + 1, target_sum + 1))
        dp[0][0] = 1
        for die in range(1, num_dice + 1):
            for sum_val in range(1, target_sum + 1):
                for face in range(1, min(faces, sum_val) + 1):
                    dp[die][sum_val] += dp[die-1][sum_val-face]
        favorable = dp[num_dice][target_sum]
    
    return favorable / total

# Bayes' theorem
def bayes_theorem(p_b_given_a, p_a, p_b):
    """P(A|B) = P(B|A) * P(A) / P(B)"""
    return (p_b_given_a * p_a) / p_b

# Example: Medical test
p_disease = 0.01  # Prior probability
p_positive_given_disease = 0.95  # Sensitivity
p_positive_given_healthy = 0.05  # False positive rate

p_healthy = 1 - p_disease
p_positive = (p_positive_given_disease * p_disease + 
              p_positive_given_healthy * p_healthy)
p_disease_given_positive = bayes_theorem(
    p_positive_given_disease, p_disease, p_positive
)

print(f"Probability of disease given positive test: {p_disease_given_positive:.4f}")

# Examples
print(f"\nBirthday problem (23 people): {birthday_prob(23):.4f}")
print(f"Dice: P(sum=7 with 2 dice) = {dice_probability(7):.4f}")
print(f"Simulation: {simulate_birthday(23):.4f}")

---

Applications in AI/ML

Key Concepts

• Bayes' Theorem: $P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)}$
• Maximum Likelihood Estimation (MLE): $\hat{\theta} = \arg\max_\theta P(\text{data}|\theta)$
• Posterior Distribution: $P(\theta|\text{data}) \propto P(\text{data}|\theta) \cdot P(\theta)$

---

Common Mistakes

---

Interview Questions

1. What is the probability of getting at least one heads in 3 coin flips?

Answer: $1 - P(\text{all tails}) = 1 - (1/2)^3 = 7/8 = 0.875$

2. Explain the difference between independent and mutually exclusive events.

Answer:

• Independent: $P(A \cap B) = P(A)P(B)$; one event doesn't affect the other.
• Mutually exclusive: $A \cap B = \emptyset$, so $P(A \cap B) = 0$; they cannot occur together.
• Note: Mutually exclusive events are generally not independent (unless one has probability 0).

3. How would you calculate the probability of being dealt a flush in poker?

Answer: A flush is 5 cards of the same suit. Total 5-card hands: $\binom{52}{5}$. Favorable: $4 \times \binom{13}{5}$. Probability: $\frac{4 \times \binom{13}{5}}{\binom{52}{5}} \approx 0.00198$.

4. What is Bayes' theorem and why is it important in ML?

Answer: Bayes' theorem: $P(A|B) = \frac{P(B|A)P(A)}{P(B)}$. It updates prior beliefs with evidence. In ML, it's used for classification (Naive Bayes), Bayesian optimization, and probabilistic models.

5. A test has 99% accuracy and 1% prevalence of disease. If you test positive, what's the probability you have the disease?

Answer: Using Bayes' theorem: $P(\text{disease}|\text{positive}) = \frac{0.99 \times 0.01}{0.99 \times 0.01 + 0.01 \times 0.99} = 0.5$ (50%).

6. How many ways can you arrange 5 books on a shelf?

Answer: $5! = 120$ permutations.

7. What's the probability of rolling a sum of 7 with two dice?

Answer: Favorable outcomes: (1,6), (2,5), (3,4), (4,3), (5,2), (6,1) = 6. Total: 36. Probability: $6/36 = 1/6 \approx 0.1667$.

---

Practice Problems

---

---

---

---

---

Probability Inequalities

---

Quick Reference

---

Cross-References

• Conditional Probability: See Conditional Probability for deeper dive into $P(A|B)$.
• Random Variables: See Random Variables for discrete and continuous distributions.
• Distributions: See Common Distributions for Bernoulli, Binomial, Poisson, etc.
• Statistical Inference: See Hypothesis Testing for applying probability in decision-making.
• Machine Learning: See Bayesian Methods for probabilistic ML models.

---