Word Embeddings: Word2Vec, GloVe, and FastText
Module: Natural Language Processing | Difficulty: Advanced
Skip-Gram Objective
Negative Sampling
GloVe (Pennington et al., 2014)
FastText (Bojanowski et al., 2017)
| Model | Analogy | Similarity | OOV | |-------|---------|-----------|-----| | Word2Vec | 75% | 0.73 | No | | GloVe | 77% | 0.74 | No | | FastText | 79% | 0.72 | Yes |
import numpy as np
from collections import Counter
class SkipGram:
def __init__(self, vocab_size, embed_dim=300, lr=0.025):
self.W_in = np.random.randn(vocab_size, embed_dim) * 0.01
self.W_out = np.random.randn(vocab_size, embed_dim) * 0.01
self.lr = lr
def train_pair(self, center, context, negative_samples):
# Forward
h = self.W_in[center]
pos_score = np.dot(self.W_out[context], h)
neg_scores = self.W_out[negative_samples] @ h
# Backward
pos_grad = (1 - np.tanh(pos_score)) * h
neg_grad = (1 - np.tanh(neg_scores))[:, None] * h
self.W_out[context] -= self.lr * pos_grad
self.W_out[negative_samples] -= self.lr * neg_grad
self.W_in[center] -= self.lr * (pos_grad * self.W_out[context] + neg_grad.sum(0))
Research Insight: Word2Vec's skip-gram with negative sampling is equivalent to factorizing the PMI matrix. This connection explains why word embeddings capture semantic relationships â they approximate the logarithm of pointwise mutual information.