Text Classification: From CNNs to Transformers
Module: Natural Language Processing | Difficulty: Advanced
TextCNN (Kim, 2014)
Filters of different sizes capture n-gram patterns.
Hierarchical Attention
BERT for Classification
Multi-Task Classification
import torch
import torch.nn as nn
class TextCNN(nn.Module):
def __init__(self, vocab_size, embed_dim=300, num_filters=100, filter_sizes=[3,4,5]):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embed_dim)
self.convs = nn.ModuleList([
nn.Conv1d(embed_dim, num_filters, fs) for fs in filter_sizes
])
self.fc = nn.Linear(num_filters * len(filter_sizes), 1)
def forward(self, x):
emb = self.embedding(x).permute(0, 2, 1)
conv_outs = [torch.relu(conv(emb)).max(dim=2)[0] for conv in self.convs]
out = torch.cat(conv_outs, dim=1)
return self.fc(out)
Research Insight: TextCNN remains competitive with transformers for short text classification because n-gram filters capture local patterns effectively. The key advantage is computational efficiency â TextCNN is 10-100x faster than BERT.