NLP Basics: Tokenization, Embeddings and Word Vectors

Natural Language Processing (NLP) bridges human language and computational understanding. This lesson covers the foundational building blocks — from raw text to dense vector representations that capture semantic meaning.

1. The NLP Pipeline

Every NLP system follows a sequential pipeline that transforms raw text into machine-understandable representations.

Key stages:

1. Raw Text — Unprocessed input (documents, sentences, tweets)
2. Tokenization — Splitting text into atomic units (tokens)
3. Normalization — Lowercasing, removing noise, stemming/lemmatization
4. Feature Extraction — Converting tokens to numerical vectors
5. Modeling — Applying statistical or neural models
6. Output — Classification, generation, translation, etc.

2. Text Preprocessing

Preprocessing cleans and normalizes text to reduce vocabulary size and noise.

2.1 Lowercasing and Noise Removal

2.2 Stopword Removal

Stopwords are high-frequency, low-information words (the, is, at, which).

2.3 Stemming vs Lemmatization

When to use which:

• Stemming: Information retrieval, search engines, when speed matters
• Lemmatization: Text analysis, chatbots, when semantic accuracy matters

3. Tokenization Strategies

Tokenization determines how text is segmented into processable units.

3.1 Word-Level Tokenization

3.2 Byte-Pair Encoding (BPE)

BPE iteratively merges the most frequent character pairs, building a subword vocabulary.

Algorithm:

1. Start with character-level vocabulary
2. Count all adjacent symbol pairs
3. Merge the most frequent pair into a new symbol
4. Repeat until desired vocabulary size is reached

3.3 WordPiece Tokenization

Used by BERT. Similar to BPE but merges pairs that maximize likelihood of the training data rather than pure frequency.

BERT tokenizer output:
"unhappiness" → ["##un", "##happi", "##ness"]
"tokenization" → ["token", "##ization"]

3.4 SentencePiece

Language-agnostic tokenization that treats input as raw Unicode, handling languages without whitespace.

Tokenizer Comparison

4. Bag of Words and TF-IDF

4.1 Bag of Words (BoW)

BoW represents documents as fixed-length vectors of word counts, ignoring order.

Limitations:

• Loses word order: "dog bites man" = "man bites dog"
• High dimensionality: vocabulary-sized sparse vectors
• No semantic information: "good" and "excellent" are unrelated

4.2 TF-IDF (Term Frequency–Inverse Document Frequency)

TF-IDF weights words by their importance within a document relative to the corpus.

5. Word Embeddings

Word embeddings map words to dense, low-dimensional vectors where geometric relationships encode semantic similarity.

5.1 Why Not One-Hot Encoding?

One-hot vectors are orthogonal — no notion of similarity:

Dense embeddings solve this by learning a continuous vector space.

5.2 Word2Vec (Mikolov et al., 2013)

Word2Vec learns embeddings by predicting context from words (or words from context).

CBOW (Continuous Bag of Words)

Predicts the center word given surrounding context words.

where v̄ = (1/2c) ∑_{j∈[-c,c], j≠0} W · x_{t+j} is the averaged context embedding.

Skip-gram

Predicts context words given the center word — the reverse of CBOW.

Negative Sampling (approximation):

5.3 GloVe (Global Vectors, Pennington et al., 2014)

GloVe combines global co-occurrence statistics with local context learning.

where X_{ij} is the co-occurrence count and f(x) is a weighting function:

Word2Vec vs GloVe

6. Embedding Properties

6.1 Word Analogies

Word embeddings capture linear relationships: king - man + woman ≈ queen.

6.2 Clustering and Semantic Groups

Embeddings form clusters where semantically related words are proximal:

Cluster 1 (royalty):    king, queen, prince, throne, crown
Cluster 2 (food):       pizza, pasta, burger, restaurant, menu
Cluster 3 (emotions):   happy, sad, angry, joyful, depressed
```python
from sklearn.manifold import TSNE
from sklearn.cluster import KMeans

words = list(glove.keys())[:5000]
vectors = np.array([glove[w] for w in words])

# Dimensionality reduction
tsne = TSNE(n_components=2, random_state=42)
coords = tsne.fit_transform(vectors)

# Clustering
kmeans = KMeans(n_clusters=10, random_state=42)
labels = kmeans.fit_predict(vectors)

# Visualize
import matplotlib.pyplot as plt
plt.scatter(coords[:, 0], coords[:, 1], c=labels, cmap='tab10', s=5)
plt.show()

6.3 Cosine Similarity

7. Sequence Representations

7.1 One-Hot Encoding

Each word is represented as a binary vector of size |V|:

Problem: For |V| = 50,000, each word is a 50K-dimensional sparse vector.

7.2 Word2Vec Averaging (Sentence Embeddings)

A simple sentence representation by averaging word vectors:

Limitations: Averaging loses word order — "dog bites man" and "man bites dog" produce identical embeddings.

7.3 Comparison of Representations

8. Complete Implementation

8.1 End-to-End NLP Pipeline

8.2 Training Word2Vec from Scratch

Key Takeaways

1. Preprocessing matters — lowercasing, stopword removal, and lemmatization significantly affect downstream performance
2. Subword tokenization (BPE, WordPiece) balances vocabulary size with OOV handling
3. TF-IDF weights words by importance: frequent in a document but rare across the corpus
4. Word2Vec learns embeddings via local context prediction; GloVe leverages global co-occurrence statistics
5. Vector arithmetic captures semantic relationships: king - man + woman ≈ queen
6. Cosine similarity measures semantic closeness in embedding space
7. Limitations of bag-of-words approaches: lose word order, syntax, and context — leading to contextual embeddings (BERT, GPT) in modern NLP

Next: Contextual Embeddings and Transformers — How BERT and GPT solve the polysemy problem with context-dependent representations.