Textual Entailment: Models and Challenges

Module: Natural Language Processing | Difficulty: Advanced

Adversarial NLI (ANLI)

Stress Tests

| Test Type | BERT | RoBERTa | Human | |-----------|------|---------|-------| | Negation | 72.3 | 81.5 | 95.2 | | Numerical | 65.1 | 73.8 | 92.1 | | Temporal | 68.4 | 76.2 | 93.5 |

HANS (Heuristic Analysis for NLI Systems)

Models rely on heuristics like lexical overlap.

Robustness

def stress_test(model, test_suite):
    results = {}
    for test_name, dataset in test_suite.items():
        correct = 0
        for batch in dataset:
            preds = model(batch['premise'], batch['hypothesis'])
            correct += (preds == batch['label']).sum().item()
        results[test_name] = correct / len(dataset)
    return results

Research Insight: NLI models learn surface-level heuristics (like word overlap) instead of true inference. ANLI shows that adversarial training improves robustness but reduces in-distribution accuracy by 3-5%. The gap between NLI and true understanding remains large.