Specialized Topics

Natural Language Processing — Teaching Computers to Read

NLP enables computers to understand, interpret, and generate human language — bridging the gap between raw text and actionable insights.

Text Preprocessing — tokenization, stemming, and lemmatization clean and normalize raw text
TF-IDF and Bag of Words — simple but effective vectorization methods for text classification
Word Embeddings — Word2Vec and GloVe capture semantic relationships between words in dense vector space

"Language is the house of being." — Martin Heidegger

Prerequisites

Before diving into NLP, you should be familiar with:

Python Programming — comfortable with strings, lists, and dictionaries
Linear Algebra — vectors, matrices, and dot products
Basic Statistics — probability, frequency distributions
Scikit-learn — basic ML pipeline and model fitting
Pandas — DataFrame manipulation for text data
Regex — regular expressions for text pattern matching

Learning Objectives

By the end of this tutorial, you will be able to:

Implement a complete NLP preprocessing pipeline
Apply tokenization, stemming, and lemmatization techniques
Convert text to numerical features using Bag of Words and TF-IDF
Understand and use word embeddings (Word2Vec, GloVe)
Build text classification models for sentiment analysis
Handle common NLP challenges like stopwords and out-of-vocabulary words
Evaluate NLP models using appropriate metrics
Apply N-grams to capture local word order

Mathematical Foundations

TF-IDF Formula

where:

(term frequency)
(inverse document frequency)

Cosine Similarity (for embeddings)

Word2Vec Skip-gram Objective

Key Formulas Reference

Essential NLP Formulas

Formula	Description
`TF(t,d) = count(t,d) / len(d)`	Term frequency in document
`IDF(t) = log(N / df(t))`	Inverse document frequency
`TF-IDF = TF × IDF`	Weighted term importance
`cos(a,b) = a·b / (‖a‖·‖b‖)`	Cosine similarity between vectors
`P(w₂\|w₁) = exp(h·v₂) / Σ exp(h·vᵢ)`	Skip-gram probability

NLP Preprocessing Pipeline

MathExample: Complete Preprocessing Pipeline

import nltk
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, WordNetLemmatizer
from nltk.tokenize import word_tokenize
import re

nltk.download('punkt')
nltk.download('stopwords')
nltk.download('wordnet')

def preprocess_text(text):
    # Lowercase
    text = text.lower()
    
    # Remove special characters and numbers
    text = re.sub(r'[^a-zA-Z\s]', '', text)
    
    # Tokenize
    tokens = word_tokenize(text)
    
    # Remove stopwords
    stop_words = set(stopwords.words('english'))
    tokens = [t for t in tokens if t not in stop_words]
    
    # Lemmatize
    lemmatizer = WordNetLemmatizer()
    tokens = [lemmatizer.lemmatize(t) for t in tokens]
    
    return ' '.join(tokens)

# Example
text = "The cats are running quickly towards the beautiful flowers!"
print(preprocess_text(text))
# Output: "cat running quickly towards beautiful flower"

Bag of Words and TF-IDF

Word Embeddings Space

MathExample: TF-IDF from Scratch

import numpy as np
from collections import Counter

def compute_tfidf(documents):
    # Build vocabulary
    vocabulary = list(set(word for doc in documents for word in doc.split()))
    vocab_idx = {word: i for i, word in enumerate(vocabulary)}
    
    # Compute TF
    tf_matrix = np.zeros((len(documents), len(vocabulary)))
    for i, doc in enumerate(documents):
        word_counts = Counter(doc.split())
        for word, count in word_counts.items():
            tf_matrix[i, vocab_idx[word]] = count / len(doc.split())
    
    # Compute IDF
    n_docs = len(documents)
    idf = np.zeros(len(vocabulary))
    for j, word in enumerate(vocabulary):
        doc_freq = sum(1 for doc in documents if word in doc.split())
        idf[j] = np.log(n_docs / doc_freq)
    
    # TF-IDF
    tfidf = tf_matrix * idf
    return tfidf, vocabulary

# Example
docs = ["I love machine learning", "I love dogs", "Machine learning is great"]
tfidf, vocab = compute_tfidf(docs)
print("Vocabulary:", vocab)
print("TF-IDF matrix:\n", tfidf)

Word Embeddings

MathNote: Word Embeddings Best Practices

N-grams and Local Word Order

Text Classification

MathExample: Sentiment Analysis with Multiple Models

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.svm import LinearSVC
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score
from sklearn.datasets import fetch_20newsgroups

# Load data (using 2 categories as sentiment proxy)
categories = ['rec.sport.baseball', 'sci.space']
data = fetch_20newsgroups(subset='train', categories=categories)

# Define models
models = {
    'Naive Bayes': Pipeline([
        ('tfidf', TfidfVectorizer(max_features=5000)),
        ('clf', MultinomialNB())
    ]),
    'Logistic Regression': Pipeline([
        ('tfidf', TfidfVectorizer(max_features=5000)),
        ('clf', LogisticRegression(max_iter=1000))
    ]),
    'SVM': Pipeline([
        ('tfidf', TfidfVectorizer(max_features=5000)),
        ('clf', LinearSVC())
    ])
}

# Compare models
for name, model in models.items():
    scores = cross_val_score(model, data.data, data.target, cv=5, scoring='accuracy')
    print(f"{name}: {scores.mean():.4f} (+/- {scores.std():.4f})")

# Output:
# Naive Bayes: 0.9823 (+/- 0.0089)
# Logistic Regression: 0.9912 (+/- 0.0056)
# SVM: 0.9891 (+/- 0.0067)

Real-World Applications

1. Sentiment Analysis

Analyzing customer reviews, social media posts, and feedback to understand public opinion about products and services.

2. Spam Detection

Email filtering using text classification to identify and filter out spam messages.

3. Named Entity Recognition (NER)

Extracting names, organizations, locations, and dates from text for information extraction.

4. Machine Translation

Converting text from one language to another using sequence-to-sequence models.

5. Question Answering

Building systems that can read documents and answer questions about their content.

6. Text Summarization

Automatically generating concise summaries of long documents or articles.

Common Mistakes & How to Avoid Them

1. Ignoring Text Preprocessing

Raw text with special characters, case differences, and stopwords hurts model performance. Always preprocess.

2. Using BoW Without TF-IDF

Raw word counts treat all words equally. TF-IDF downweights common words and emphasizes distinctive terms.

3. Not Handling Imbalanced Classes

Sentiment datasets are often imbalanced. Use class weights, oversampling, or undersampling.

4. Ignoring Word Order Completely

Bag of Words loses all word order. Consider N-grams or embeddings for context-dependent tasks.

5. Not Using Cross-Validation

Text classification can have high variance due to vocabulary. Always use k-fold cross-validation.

6. Overfitting to Small Datasets

Deep learning models need lots of data. For small datasets, traditional ML with TF-IDF often works better.

Interview Questions

Q1: What's the difference between stemming and lemmatization?

A: Stemming applies crude rules to chop word endings (running → run, ran → run). Lemmatization uses vocabulary and morphological analysis to return the dictionary form (better → good, ran → run). Lemmatization is more accurate but slower.

Q2: Why is TF-IDF better than simple word counts?

A: TF-IDF downweights words that appear frequently across all documents (like "the", "is") and upweights words that are distinctive to specific documents. This gives more informative features.

Q3: How do Word2Vec embeddings capture semantic meaning?

A: Word2Vec learns to predict words from their context (or vice versa) using neural networks. Words appearing in similar contexts end up with similar vector representations, capturing semantic relationships.

Q4: What is the curse of dimensionality in NLP?

A: Vocabulary can be huge (100K+ words), making one-hot or BoW vectors very sparse and high-dimensional. Techniques like TF-IDF, embeddings, and dimensionality reduction help combat this.

Q5: How do you handle out-of-vocabulary words?

A: Options include: using <UNK> token, subword tokenization (BPE, WordPiece), character-level embeddings, or FastText which handles unknown words by composing subword embeddings.

Q6: When would you use Naive Bayes vs SVM for text classification?

A: Naive Bayes is fast, works well with small data, and provides probabilistic outputs. SVM often achieves higher accuracy with sufficient data. Naive Bayes is a good baseline; SVM for better performance.

Q7: How do you evaluate NLP models?

A: Common metrics: accuracy, precision, recall, F1-score (for classification), BLEU (for translation), ROUGE (for summarization), perplexity (for language models). Use stratified k-fold cross-validation.

Practice Exercise

Exercise: Build a Sentiment Analyzer

Objective: Build a complete sentiment analysis pipeline.

Dataset: Use sklearn.datasets.fetch_20newsgroups with categories ['rec.sport.baseball', 'sci.space'] as a proxy for positive/negative sentiment.

Tasks:

Preprocess the text — apply lowercase, remove special characters, remove stopwords
Create TF-IDF features with different parameters:
- Unigrams only
- Unigrams + bigrams
- With max_features = [1000, 5000, 10000]
Train and compare:
- Naive Bayes
- Logistic Regression
- Linear SVM
Evaluate using 5-fold cross-validation
Analyze which features and model perform best

Bonus: Try using pre-trained Word2Vec embeddings as features instead of TF-IDF.

Comparison Table

NLP Feature Extraction Methods Comparison

Feature	Bag of Words	TF-IDF	Word Embeddings
Representation	Sparse counts	Sparse weighted	Dense vectors
Semantic Meaning	None	Limited	Rich
Word Order	Lost	Lost (unless N-grams)	Context-dependent
Dimensionality	High (vocabulary size)	High (vocabulary size)	Low (50-300)
Computational Cost	Low	Low	Medium
Best For	Simple classification	Text classification	Semantic tasks

Key Takeaways

What to Learn Next

-> Transformers Learn the self-attention architecture that revolutionized NLP and powers modern AI.

-> BERT Master encoder-only transformers for text classification, NER, and question answering.

-> GPT Architecture Understand decoder-only transformers that power autoregressive text generation.

-> Naive Bayes Learn the simple probabilistic classifier often used as a strong NLP baseline.

-> RNN and LSTM Explore sequential models that were the dominant NLP approach before transformers.

-> GANs Discover generative adversarial networks for text generation and style transfer.

NLP Fundamentals — Text Processing, Embeddings and Classification