Supervised Learning

Probability-Based Classification — Fast, Simple, and Surprisingly Effective

Naive Bayes applies Bayes theorem with a strong independence assumption. Despite this simplification, it performs remarkably well on text classification, spam filtering, and medical diagnosis.

Bayes Theorem — Converting prior beliefs with evidence
Conditional Independence — The "naive" assumption
Likelihood Models — Gaussian, Multinomial, Bernoulli

"In God we trust; all others must bring data." — W. Edwards Deming

Prerequisites

Before diving in, make sure you're comfortable with:

Basic Probability — Conditional probability, joint probability, independence
Bayes Theorem — Prior, posterior, likelihood (introduced here)
Python — NumPy basics, scikit-learn usage
Text Classification — Bag of words, TF-IDF concepts

Learning Objectives

After completing this tutorial, you will be able to:

Apply Bayes theorem to derive the Naive Bayes classifier
Explain the conditional independence assumption and its implications
Implement Gaussian, Multinomial, and Bernoulli Naive Bayes variants
Handle continuous features using Gaussian density estimation
Understand Laplace smoothing and why it prevents zero probabilities
Know when Naive Bayes works well and when it fails

Naive Bayes Classifier — Complete Guide

Naive Bayes is a probabilistic classifier that uses Bayes theorem with a "naive" assumption of feature independence.

Bayes Theorem

Why "Naive"? Independence Assumption

Naive Bayes Variants

Gaussian Naive Bayes

from sklearn.naive_bayes import GaussianNB
from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score

iris = load_iris()
gnb = GaussianNB()
scores = cross_val_score(gnb, iris.data, iris.target, cv=5)
print(f"GaussianNB accuracy: {scores.mean():.3f} ± {scores.std():.3f}")

gnb.fit(iris.data, iris.target)
print(f"Class priors: {gnb.class_prior_}")
print(f"Class means:\n{gnb.theta_}")
print(f"Class variances:\n{gnb.var_}")

Multinomial Naive Bayes

from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.pipeline import make_pipeline

train_docs = ["win free money now", "meeting tomorrow at ten", "free prize claim", "project deadline friday"]
train_labels = [1, 0, 1, 0]  # 1=spam, 0=not spam

model = make_pipeline(CountVectorizer(), MultinomialNB(alpha=1.0))
model.fit(train_docs, train_labels)
test_docs = ["free money winner", "team meeting agenda"]
print(f"Predictions: {model.predict(test_docs)}")
print(f"Probabilities:\n{model.predict_proba(test_docs)}")

Bernoulli Naive Bayes

Laplace Smoothing

Complete Example: Email Spam Classification

import numpy as np
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import classification_report, confusion_matrix

# Sample email dataset
emails = [
    "Win a free iPhone now! Click here!",
    "Meeting scheduled for tomorrow at 3pm",
    "Congratulations! You've won $1000 prize",
    "Please review the attached document",
    "Free money transfer available immediately",
    "Team lunch Friday at noon",
    "Claim your free vacation package now",
    "Project status update needed by EOD",
    "Exclusive deal just for you! Act fast!",
    "Quarterly report ready for review",
    "You have been selected for a cash prize!",
    "Can we reschedule our 1-on-1 meeting?",
    "Free trial offer expires today!",
    "Budget review meeting next Tuesday",
    "Make money fast with this simple trick",
    "Action required: sign the contract",
]
labels = [1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0]  # 1=spam

# TF-IDF features
vectorizer = TfidfVectorizer(stop_words='english', max_features=1000)
X = vectorizer.fit_transform(emails)
y = np.array(labels)

# Train with cross-validation
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
nb = MultinomialNB(alpha=1.0)
nb.fit(X_train, y_train)
print(f"Accuracy: {nb.score(X_test, y_test):.3f}")

# Top features per class
feature_names = vectorizer.get_feature_names_out()
for i, label in enumerate(['Ham', 'Spam']):
    top_indices = nb.feature_log_prob_[i].argsort()[-10:][::-1]
    print(f"\nTop features for {label}:")
    for idx in top_indices:
        print(f"  {feature_names[idx]}: {np.exp(nb.feature_log_prob_[i][idx]):.4f}")

Mathematical Derivation

Real-World Applications

Spam Filtering

The classic application — Naive Bayes was one of the first successful spam filters (Paul Graham, 2002). Words like "free", "winner", "click" have high likelihood in spam class. Fast training and prediction make it ideal for real-time email filtering.

Sentiment Analysis

Multinomial Naive Bayes classifies movie reviews as positive/negative based on word frequencies. Despite the independence assumption (words are clearly not independent), it achieves ~85% accuracy on standard benchmarks — competitive with much more complex models.

Medical Diagnosis

Naive Bayes helps diagnose diseases based on symptoms. Each symptom is treated independently: P(fever | flu), P(cough | flu), etc. The speed enables real-time clinical decision support.

Text Classification

News categorization, language detection, topic assignment — Naive Bayes excels at any text classification task where features are word counts or TF-IDF scores. It trains in seconds on millions of documents.

Recommendation Systems

Bernoulli Naive Bayes can predict user preferences based on item interaction patterns, treating each item as a binary feature.

Common Mistakes & How to Avoid Them

Mistake 1: Not applying smoothing

Problem: Zero probabilities when a word never appears in a class — entire posterior collapses to zero
Solution: Always use alpha=1.0 (Laplace smoothing), especially with small datasets

Mistake 2: Using Gaussian Naive Bayes on non-Gaussian data

Problem: Gaussian assumption is wrong for count data, binary data, or heavily skewed distributions
Solution: Match variant to data type — Gaussian for continuous, Multinomial for counts, Bernoulli for binary

Mistake 3: Ignoring feature correlations

Problem: Dependent features get "double-counted," distorting probabilities
Solution: Remove highly correlated features, or use dimensionality reduction (PCA) before Naive Bayes

Mistake 4: Using Naive Bayes when probabilities matter

Problem: Naive Bayes produces poorly calibrated probabilities (often overconfident)
Solution: Use Naive Bayes for ranking/classification, not for probability estimation. Calibrate with Platt scaling if probabilities are needed.

Mistake 5: Feeding continuous features directly to MultinomialNB

Problem: MultinomialNB expects non-negative counts, not continuous values
Solution: Use GaussianNB for continuous features, or discretize/bin continuous features first

Interview Questions

Q1: Why is Naive Bayes called "naive"? A: Because it assumes all features are conditionally independent given the class. In reality, features are often correlated (e.g., "free" and "money" in spam). Despite this, the classifier often works well because it only needs to rank classes correctly, not estimate exact probabilities.

Q2: When does Naive Bayes work well despite the independence assumption? A: When the ranking of classes is preserved even though probabilities are miscalibrated. This happens when: (1) correlated features are consistently correlated across classes, (2) the task requires classification not probability estimation, (3) there are many features that provide independent signals.

Q3: What is Laplace smoothing and why is it necessary? A: Laplace smoothing adds (typically 1) to all feature counts to prevent zero probabilities. Without it, if a feature value never occurs with a class in training data, the entire posterior becomes zero — making the model unable to classify any instance with that feature.

Q4: What's the difference between Gaussian, Multinomial, and Bernoulli Naive Bayes? A: Gaussian: continuous features, models P(x|C) as Gaussian distribution. Multinomial: count data (e.g., word frequencies), models P(x|C) as multinomial distribution. Bernoulli: binary features (present/absent), models P(x=1|C) as Bernoulli distribution. Choose based on your data type.

Q5: How does Naive Bayes handle high-dimensional data like text? A: Excellent — it's one of the best algorithms for high-dimensional sparse data. The independence assumption makes training O(Nd) instead of exponential in d. With thousands of features (words), it trains in seconds and generalizes well due to low variance.

Q6: Why does Naive Bayes often outperform more complex models on small datasets? A: Because of the strong independence assumption (high bias, low variance). On small datasets, complex models overfit (high variance), while Naive Bayes' simplicity provides regularization. As data grows, more complex models may overtake it.

Q7: Can Naive Bayes be used for regression? A: No — Naive Bayes is inherently a classifier. For continuous target variables, you'd need Gaussian Naive Bayes applied to discretized targets, or use a different algorithm entirely (linear regression, Gaussian processes, etc.).

Practice Exercise

Challenge: Text Classification Comparison

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.naive_bayes import MultinomialNB, BernoulliNB, GaussianNB
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
import numpy as np

# Load subset of 20 newsgroups (4 categories for speed)
categories = ['sci.space', 'rec.sport.baseball', 'comp.graphics', 'talk.politics.misc']
data = fetch_20newsgroups(subset='all', categories=categories, remove=('headers', 'footers'))

# Different vectorizers for different NB variants
tfidf_vec = TfidfVectorizer(stop_words='english', max_features=5000)
count_vec = CountVectorizer(stop_words='english', max_features=5000)

X_tfidf = tfidf_vec.fit_transform(data.data)
X_count = count_vec.fit_transform(data.data)
X_dense = X_tfidf.toarray()  # For GaussianNB

# Compare classifiers
classifiers = {
    'MultinomialNB (TF-IDF)': make_pipeline(TfidfVectorizer(stop_words='english', max_features=5000), MultinomialNB()),
    'MultinomialNB (Counts)': make_pipeline(CountVectorizer(stop_words='english', max_features=5000), MultinomialNB()),
    'BernoulliNB': make_pipeline(CountVectorizer(stop_words='english', max_features=5000), BernoulliNB()),
    'LogisticRegression': make_pipeline(TfidfVectorizer(stop_words='english', max_features=5000), LogisticRegression(max_iter=1000)),
}

print("5-Fold Cross-Validation Results:")
print("-" * 55)
for name, clf in classifiers.items():
    scores = cross_val_score(clf, data.data, data.target, cv=5)
    print(f"{name:30s} {scores.mean():.3f} ± {scores.std():.3f}")

Bonus challenges:

Plot the effect of vocabulary size on accuracy for each NB variant
Implement Naive Bayes from scratch without scikit-learn
Add Laplace smoothing with different α values and plot accuracy vs α

Comparison Table

Naive Bayes Variants Comparison

Variant	Feature Type	Distribution	Best For
GaussianNB	Continuous	Normal	Iris, medical measurements
MultinomialNB	Counts / Frequencies	Multinomial	Text classification, TF-IDF
BernoulliNB	Binary (0/1)	Bernoulli	Document presence/absence
ComplementNB	Counts / TF-IDF	Complement	Imbalanced text classification

Key Formulas Reference

Formula	Expression	Context
Bayes Theorem		Foundation of Naive Bayes
Independence Assumption		The "naive" part
Log-posterior		Numerically stable computation
Gaussian Likelihood		GaussianNB
Multinomial Likelihood		MultinomialNB with smoothing

Key Takeaways

What to Learn Next

-> Support Vector Machines Maximum-margin classifiers for when Naive Bayes isn't enough.

-> Logistic Regression Probabilistic linear classifier that often outperforms Naive Bayes.

-> Text Classification Deep Dive Advanced NLP techniques beyond bag-of-words.

Naive Bayes Classifier — Complete Guide with Visualizations