Neural Machine Translation: From Attention to Modern Systems
Module: Natural Language Processing | Difficulty: Advanced
Transformer for MT
Beam Search
BLEU Score
where is the brevity penalty.
chrF
Character n-gram F-score.
import numpy as np
from collections import Counter
def bleu_score(reference, hypothesis, max_n=4):
precisions = []
for n in range(1, max_n + 1):
ref_ngrams = Counter([tuple(reference[i:i+n]) for i in range(len(reference)-n+1)])
hyp_ngrams = Counter([tuple(hypothesis[i:i+n]) for i in range(len(hypothesis)-n+1)])
clipped = sum(min(count, ref_ngrams[ng]) for ng, count in hyp_ngrams.items())
total = max(sum(hyp_ngrams.values()), 1)
precisions.append(clipped / total)
bp = min(1, np.exp(1 - len(reference) / max(len(hypothesis), 1)))
return bp * np.exp(np.mean(np.log([p + 1e-10 for p in precisions])))
Research Insight: BLEU correlates poorly with human judgment for short sentences and creative translations. chrF and COMET are better evaluation metrics because they capture character-level similarity and use learned representations respectively.