Multilingual NLP: Cross-lingual Transfer and Zero-Shot Learning

Module: Natural Language Processing | Difficulty: Advanced

Multilingual BERT

Cross-Lingual Transfer

Language-Agnostic Representations

Zero-Shot Transfer

Train on English, evaluate on other languages.

| Language | mBERT | XLM-R | Difference | |----------|-------|-------|------------| | English | 92.5 | 93.1 | +0.6 | | German | 85.3 | 88.2 | +2.9 | | Chinese | 78.1 | 83.5 | +5.4 | | Arabic | 76.2 | 81.8 | +5.6 |

import torch
import torch.nn as nn

class MultilingualClassifier(nn.Module):
    def __init__(self, xlm_roberta, n_classes):
        super().__init__()
        self.encoder = xlm_roberta
        self.classifier = nn.Linear(768, n_classes)
    def forward(self, input_ids, attention_mask, lang_ids=None):
        outputs = self.encoder(input_ids, attention_mask=attention_mask)
        cls_output = outputs.last_hidden_state[:, 0, :]
        return self.classifier(cls_output)

Research Insight: Cross-lingual transfer works because multilingual models learn language-agnostic representations. XLM-R's improvement over mBERT comes from training on 100x more data and using a better objective (MLM vs MLM+NSP).