BERT and Variants: Masked Language Modeling
Module: Natural Language Processing | Difficulty: Advanced
BERT Pre-Training
Masked Language Modeling (MLM):
Next Sentence Prediction (NSP):
RoBERTa Improvements
- Remove NSP
- Dynamic masking
- Larger batches, more data
ALBERT
- Factorized embedding:
- Cross-layer parameter sharing
Model Comparison
| Model | Params | Training Data | GLUE |
|---|---|---|---|
| BERT-base | 110M | 16GB | 79.6 |
| BERT-large | 340M | 16GB | 82.1 |
| RoBERTa | 355M | 160GB | 88.9 |
| ALBERT-xxlarge | 235M | 16GB | 90.4 |
import torch
import torch.nn as nn
class BertForMLM(nn.Module):
def __init__(self, vocab_size=30522, d_model=768, nhead=12, num_layers=12):
super().__init__()
self.embed = nn.Embedding(vocab_size, d_model)
self.pos_embed = nn.Embedding(512, d_model)
encoder_layer = nn.TransformerEncoderLayer(d_model, nhead, d_model*4, batch_first=True)
self.transformer = nn.TransformerEncoder(encoder_layer, num_layers)
self.head = nn.Linear(d_model, vocab_size)
def forward(self, input_ids, attention_mask=None):
positions = torch.arange(input_ids.size(1), device=input_ids.device).unsqueeze(0)
x = self.embed(input_ids) + self.pos_embed(positions)
x = self.transformer(x, src_key_padding_mask=attention_mask)
return self.head(x)
Research Insight: BERT's MLM objective learns bidirectional representations, making it superior to left-to-right models for understanding tasks. However, the [MASK] token creates a pretrain-finetune discrepancy, which XLNet addresses with permutation language modeling.