Natural Language Inference: Entailment and Contradiction
Module: Natural Language Processing | Difficulty: Advanced
NLI Task
Models
Decomposable Attention
MNLI Results
| Model | Matched | Mismatched | |-------|---------|------------| | ESIM | 75.1 | 75.6 | | BERT | 84.6 | 83.4 | | RoBERTa | 90.2 | 89.6 |
import torch
import torch.nn as nn
class NLIModel(nn.Module):
def __init__(self, bert_model, n_classes=3):
super().__init__()
self.bert = bert_model
self.classifier = nn.Sequential(
nn.Linear(768*4, 512), nn.ReLU(),
nn.Dropout(0.1), nn.Linear(512, n_classes))
def forward(self, premise_ids, hypothesis_ids, attention_mask):
p_out = self.bert(premise_ids, attention_mask=attention_mask)
h_out = self.bert(hypothesis_ids, attention_mask=attention_mask)
p_cls = p_out.last_hidden_state[:, 0]
h_cls = h_out.last_hidden_state[:, 0]
combined = torch.cat([p_cls, h_cls, p_cls*h_cls, p_cls-h_cls], dim=-1)
return self.classifier(combined)
Research Insight: NLI is a good proxy for many NLU tasks because it requires understanding of semantic relationships. Training on MNLI improves performance on other tasks by 2-5%, showing that inference knowledge transfers well.