Language Model Evaluation: Benchmetrics and Analysis

Module: Natural Language Processing | Difficulty: Advanced

Perplexity

GLUE Benchmark

| Task | Metric | BERT | RoBERTa | GPT-3 | |------|--------|------|---------|-------| | MNLI | Acc | 84.6 | 90.2 | 78.3 | | QQP | F1 | 71.2 | 92.1 | 71.3 | | SST-2 | Acc | 93.5 | 96.4 | 94.6 |

Emergent Evaluation

• Chain-of-thought reasoning
• In-context learning
• Zero-shot generalization

Evaluation Challenges

1. Benchmark contamination
2. Memorization vs understanding
3. Robustness to perturbations

import numpy as np

def perplexity(model, tokenizer, text):
    encodings = tokenizer(text, return_tensors='pt')
    input_ids = encodings.input_ids
    with torch.no_grad():
        outputs = model(input_ids, labels=input_ids)
        loss = outputs.loss
    return np.exp(loss.item())

def evaluate_glue(model, dataset, task):
    correct = 0
    total = 0
    for batch in dataset:
        logits = model(batch['input_ids'], batch['attention_mask'])
        predictions = logits.argmax(dim=-1)
        correct += (predictions == batch['labels']).sum().item()
        total += len(batch['labels'])
    return correct / total

Research Insight: GPT-4's performance on standardized tests (bar exam, SAT) suggests that language models can generalize beyond their training data. However, the mechanisms behind this generalization remain poorly understood.