Conditional Probability

Why It Matters

In real life, we rarely reason about events in isolation. We constantly update our beliefs based on new information. What is the probability that it will rain, given that the sky is cloudy? Conditional probability formalizes this intuition. It transforms vague guesses into rigorous, calculable statements.

Conditional Probability

Properties of Conditional Probability

Conditional probability is itself a probability measure — it satisfies all axioms of probability:

1. Non-negativity: P(A|B) ≥ 0 for all events A
2. Normalization: P(Ω|B) = 1 (the entire sample space given B has probability 1)
3. Additivity: For disjoint events A₁, A₂: P(A₁ ∪ A₂ | B) = P(A₁|B) + P(A₂|B)

This means every theorem and rule of probability applies within the conditioned space.

Intuitive Interpretation

Multiplication Rule

Rearranging the definition of conditional probability yields the multiplication rule — a powerful tool for computing joint probabilities.

Extended Multiplication Rule (Chain Rule)

For multiple events A₁, A₂, ..., Aₙ:

Independence

Pairwise vs. Mutual Independence

Bayes' Theorem

Bayesian Reasoning Framework

Law of Total Probability

This law allows us to decompose complex probability calculations into manageable pieces by considering all possible scenarios that could lead to event A.

Continuous Version

For continuous random variables with probability density f_X(x) and conditional density f_{Y|X}(y|x):

Medical Testing Example

This is the classic "base rate fallacy" example that demonstrates why conditional probability matters.

Setup

• Disease prevalence: P(Disease) = 0.01 (1% of population)
• Test sensitivity: P(Positive|Disease) = 0.95 (95% true positive rate)
• Test specificity: P(Negative|Healthy) = 0.95 (95% true negative rate)
• False positive rate: P(Positive|Healthy) = 0.05

What is P(Disease | Positive)?

What Happens with Higher Prevalence?

Repeated Testing

If someone tests positive twice independently, the posterior updates:

P(Disease | 2 positives) = P(2 pos|Disease)·P(Disease) / P(2 pos) = (0.95² × 0.01) / (0.95² × 0.01 + 0.05² × 0.99) ≈ 0.789 (78.9%)

Python Implementation

import numpy as np
from collections import defaultdict

def conditional_probability(joint_count, condition_count):
    """Calculate P(A|B) from counts."""
    if condition_count == 0:
        return 0.0
    return joint_count / condition_count

# --- Medical Testing Example ---
def medical_test():
    p_disease = 0.01
    p_pos_given_disease = 0.95
    p_pos_given_healthy = 0.05

    # Law of Total Probability
    p_positive = (p_pos_given_disease * p_disease +
                  p_pos_given_healthy * (1 - p_disease))

    # Bayes' Theorem
    p_disease_given_pos = (p_pos_given_disease * p_disease) / p_positive

    print(f"P(Positive) = {p_positive:.4f}")
    print(f"P(Disease | Positive) = {p_disease_given_pos:.4f}")
    return p_disease_given_pos

# --- Chain Rule: Drawing Without Replacement ---
def chain_rule_example():
    """Probability of drawing 3 aces in a row."""
    p = (4/52) * (3/51) * (2/50)
    print(f"P(3 aces) = {p:.6f}")
    return p

# --- Bayesian Spam Filter ---
def spam_filter():
    """Simple Bayesian spam classifier."""
    # Priors
    p_spam = 0.3
    p_ham = 0.7

    # Word likelihoods: P(word | spam), P(word | ham)
    word_likelihoods = {
        'free':  {'spam': 0.8, 'ham': 0.1},
        'money': {'spam': 0.6, 'ham': 0.05},
        'hello': {'spam': 0.1, 'ham': 0.5},
    }

    # Classify message with words ['free', 'money']
    words = ['free', 'money']
    p_email_spam = p_spam
    p_email_ham = p_ham

    for word in words:
        p_email_spam *= word_likelihoods[word]['spam']
        p_email_ham *= word_likelihoods[word]['ham']

    # Normalize
    total = p_email_spam + p_email_ham
    p_spam_given_words = p_email_spam / total

    print(f"P(Spam | 'free', 'money') = {p_spam_given_words:.4f}")
    return p_spam_given_words

# --- Independence Check ---
def check_independence(p_a, p_b, p_a_and_b, tolerance=1e-9):
    """Check if two events are independent."""
    independent = abs(p_a_and_b - p_a * p_b) < tolerance
    print(f"P(A)={p_a}, P(B)={p_b}, P(A∩B)={p_a_and_b}")
    print(f"Independent: {independent}")
    return independent

# --- Monte Carlo Simulation ---
def simulate_conditional(n_trials=1_000_000):
    """Monte Carlo estimation of P(A|B)."""
    # Roll two dice. A = sum > 7, B = first die > 3
    first = np.random.randint(1, 7, n_trials)
    second = np.random.randint(1, 7, n_trials)
    total = first + second

    b_mask = first > 3
    a_and_b = (total > 7) & b_mask

    p_b = b_mask.mean()
    p_a_given_b = a_and_b[b_mask].mean()

    print(f"Simulated P(A|B) = {p_a_given_b:.4f}")
    print(f"Analytical P(A|B) = {15/36 / (2/3):.4f}")
    return p_a_given_b

if __name__ == "__main__":
    print("=== Medical Test ===")
    medical_test()
    print("\n=== Chain Rule ===")
    chain_rule_example()
    print("\n=== Spam Filter ===")
    spam_filter()
    print("\n=== Independence Check ===")
    check_independence(0.3, 0.4, 0.12)   # Independent
    check_independence(0.3, 0.4, 0.10)   # Not independent

Applications in AI/ML

1. Naive Bayes Classifier

Spam Filtering Example:

For an email with words w₁, w₂, ..., wₙ:

• P(Spam | w₁, w₂, ..., wₙ) ∝ P(Spam) · P(w₁|Spam) · P(w₂|Spam) · ... · P(wₙ|Spam)
• P(Ham | w₁, w₂, ..., wₙ) ∝ P(Ham) · P(w₁|Ham) · P(w₂|Ham) · ... · P(wₙ|Ham)

Classify as whichever class has the higher posterior.

2. Bayesian Inference in Machine Learning

Bayesian methods treat model parameters as random variables with distributions:

• Prior: P(θ) — beliefs about parameters before seeing data
• Likelihood: P(D|θ) — how well parameters explain the data
• Posterior: P(θ|D) ∝ P(D|θ) · P(θ) — updated beliefs after seeing data

Applications include Bayesian optimization for hyperparameter tuning, Gaussian processes, and Bayesian neural networks.

3. Hidden Markov Models

HMMs use conditional probability chains for sequence modeling:

P(x₁, x₂, ..., xₙ, z₁, z₂, ..., zₙ) = P(z₁) ∏ P(xᵢ|zᵢ) · P(zᵢ|zᵢ₋₁)

Used in speech recognition, bioinformatics, and natural language processing.

4. Conditional Random Fields

CRFs model P(labels | observations) directly, avoiding the independence assumption of HMMs while still using the conditional probability framework.

5. Medical AI and Diagnosis

Bayesian networks represent diseases, symptoms, and test results as nodes in a directed graph. Conditional probability tables at each node enable probabilistic reasoning about patient diagnoses.

Common Mistakes

Interview Questions

Q1: Explain the difference between P(A|B) and P(B|A). Give a real-world example.

Q2: Two coins are flipped. Are the outcomes independent? How would you test this?

Q3: A test has 99% sensitivity and 99% specificity. Disease prevalence is 0.1%. What is P(Disease|Positive)?

Q4: Under what conditions is P(A|B) = P(A)? What does this mean?

Q5: How is Bayes' theorem used in spam filtering?

Q6: What is the Monty Hall problem and how does conditional probability explain it?

Practice Problems

Problem 1: Drawing Cards

Problem 2: Conditional Probability from a Table

Problem 3: Bayes' Theorem Application

Problem 4: Independence Verification

Problem 5: Disease Screening

Quick Reference

Bayesian Update Cycle: Prior -> Observe Evidence -> Compute Likelihood -> Apply Bayes -> Posterior -> (Posterior becomes new Prior)

Cross-References

• Basic Probability: See 031-probability-basics.mdx for foundational concepts
• Probability Distributions — continuous and discrete distributions
• Joint Probability — joint distributions and marginals
• Random Variables — random variable fundamentals
• Combinatorics: See 030-probability-combinatorics.mdx for counting techniques essential to probability calculations
• Statistics Inference: See 040-statistics-inference.mdx for connecting probability to statistical inference
• Machine Learning: Conditional probability is fundamental to Naive Bayes classifiers, Bayesian optimization, and probabilistic graphical models