🎉 75% of content is free forever — Unlock Premium from $10/mo →
CW
Search courses…
💼 Servicesℹ️ About✉️ ContactView Pricing Plansfrom $10

Probability Foundations

ProbabilityBasics🟢 Free Lesson

Advertisement

Probability Foundations


What is Probability


Sample Space and Events


Axioms of Probability

These axioms form the foundation of all probability theory. The third axiom extends to countably many events via sigma-additivity.


Basic Properties

Derived from the axioms, these properties are essential:

  • Complement Rule: , where is the complement of .
  • Monotonicity: If , then .
  • General Addition Rule: For any events and :
  • Bonferroni's Inequality: .
  • Boole's Inequality: (union bound).

Counting Principles

Counting techniques are fundamental for computing probabilities in discrete settings.

Extended Counting Formulas

  • Permutations with Repetition: for multiset.
  • Circular Permutations: for arranging items in a circle.
  • Stars and Bars: Number of ways to distribute identical items into distinct bins: .

Inclusion-Exclusion

Example: Two Events

Example: Three Events


Complement Rule

This rule is particularly useful for "at least one" problems:


Python Implementation

import numpy as np
from math import comb, perm, factorial
from itertools import combinations, permutations

# Basic counting
print(f"Permutations P(10,3): {perm(10, 3)}")  # 720
print(f"Combinations C(10,3): {comb(10, 3)}")  # 120

# Probability of union
def prob_union(p_a, p_b, p_ab):
    """P(A ∪ B) = P(A) + P(B) - P(A ∩ B)"""
    return p_a + p_b - p_ab

# Inclusion-Exclusion for three events
def prob_union_three(p_a, p_b, p_c, p_ab, p_ac, p_bc, p_abc):
    """P(A ∪ B ∪ C) using inclusion-exclusion"""
    return (p_a + p_b + p_c 
            - p_ab - p_ac - p_bc 
            + p_abc)

# Birthday problem
def birthday_prob(n, days=365):
    """Probability of at least one collision in n birthdays"""
    if n > days:
        return 1.0
    prob_no_collision = 1.0
    for i in range(n):
        prob_no_collision *= (days - i) / days
    return 1 - prob_no_collision

# Simulation approach
def simulate_birthday(n, days=365, trials=100000):
    """Monte Carlo simulation for birthday problem"""
    collisions = 0
    for _ in range(trials):
        birthdays = np.random.randint(0, days, size=n)
        if len(np.unique(birthdays)) < n:
            collisions += 1
    return collisions / trials

# Dice problems
def dice_probability(target_sum, num_dice=2, faces=6):
    """Probability of getting target_sum with num_dice dice"""
    # Count favorable outcomes
    favorable = 0
    total = faces ** num_dice
    
    # Brute force for small cases
    if num_dice == 2:
        for i in range(1, faces+1):
            for j in range(1, faces+1):
                if i + j == target_sum:
                    favorable += 1
    else:
        # Use dynamic programming
        dp = np.zeros((num_dice + 1, target_sum + 1))
        dp[0][0] = 1
        for die in range(1, num_dice + 1):
            for sum_val in range(1, target_sum + 1):
                for face in range(1, min(faces, sum_val) + 1):
                    dp[die][sum_val] += dp[die-1][sum_val-face]
        favorable = dp[num_dice][target_sum]
    
    return favorable / total

# Bayes' theorem
def bayes_theorem(p_b_given_a, p_a, p_b):
    """P(A|B) = P(B|A) * P(A) / P(B)"""
    return (p_b_given_a * p_a) / p_b

# Example: Medical test
p_disease = 0.01  # Prior probability
p_positive_given_disease = 0.95  # Sensitivity
p_positive_given_healthy = 0.05  # False positive rate

p_healthy = 1 - p_disease
p_positive = (p_positive_given_disease * p_disease + 
              p_positive_given_healthy * p_healthy)
p_disease_given_positive = bayes_theorem(
    p_positive_given_disease, p_disease, p_positive
)

print(f"Probability of disease given positive test: {p_disease_given_positive:.4f}")

# Examples
print(f"\nBirthday problem (23 people): {birthday_prob(23):.4f}")
print(f"Dice: P(sum=7 with 2 dice) = {dice_probability(7):.4f}")
print(f"Simulation: {simulate_birthday(23):.4f}")

Applications in AI/ML

Key Concepts

  • Bayes' Theorem:
  • Maximum Likelihood Estimation (MLE):
  • Posterior Distribution:

Common Mistakes

MistakeExplanationCorrect Approach
Assuming events are independent only if independentCheck if one event affects the other
Confusing permutations and combinationsOrder matters for permutationsUse combinations when order doesn't matter
Ignoring the complement rule"At least one" problems are easier with complementsAlways consider
Double-counting in inclusion-exclusionOverlapping probabilities counted multiple timesUse the full inclusion-exclusion formula
Forgetting conditional probability in generalApply Bayes' theorem when conditioning
Assuming equally likely outcomesNot all sample spaces have uniform probabilitiesUse the definition of probability, not symmetry

Interview Questions

1. What is the probability of getting at least one heads in 3 coin flips?

Answer:

2. Explain the difference between independent and mutually exclusive events.

Answer:

  • Independent: ; one event doesn't affect the other.
  • Mutually exclusive: , so ; they cannot occur together.
  • Note: Mutually exclusive events are generally not independent (unless one has probability 0).

3. How would you calculate the probability of being dealt a flush in poker?

Answer: A flush is 5 cards of the same suit. Total 5-card hands: . Favorable: . Probability: .

4. What is Bayes' theorem and why is it important in ML?

Answer: Bayes' theorem: . It updates prior beliefs with evidence. In ML, it's used for classification (Naive Bayes), Bayesian optimization, and probabilistic models.

5. A test has 99% accuracy and 1% prevalence of disease. If you test positive, what's the probability you have the disease?

Answer: Using Bayes' theorem: (50%).

6. How many ways can you arrange 5 books on a shelf?

Answer: permutations.

7. What's the probability of rolling a sum of 7 with two dice?

Answer: Favorable outcomes: (1,6), (2,5), (3,4), (4,3), (5,2), (6,1) = 6. Total: 36. Probability: .


Practice Problems






Probability Inequalities


Quick Reference

ConceptFormulaNotes
Sample SpaceSet of all outcomes
EventSubset of outcomes
ComplementProbability event doesn't occur
Addition RuleFor any events
Multiplication RuleChain rule
IndependenceNo influence
Mutually ExclusiveCannot co-occur
Bayes' TheoremUpdate beliefs
PermutationsOrder matters
CombinationsOrder doesn't matter
Inclusion-ExclusionAvoid double-counting
Law of Total ProbabilityPartition of sample space

Cross-References


Need Expert Mathematics Help?

Get personalized tutoring, project support, or professional consulting.

Advertisement