LLM for Translation

Machine translation has evolved from rule-based systems to statistical methods to neural approaches. LLMs represent the latest evolution, offering unprecedented multilingual capabilities through scale, transfer learning, and instruction following.

Translation Formulation

The model learns a conditional distribution over target tokens given the source sequence. During inference, the model generates translations by sampling from this conditional distribution.

Multilingual Models

Multilingual LLMs are trained on text from multiple languages simultaneously, enabling cross-lingual transfer and zero-shot translation.

Language Coverage

Translation Quality Metrics

Evaluating translation quality requires both automatic metrics and human evaluation.

BLEU Score

COMET Score

COMET is a neural evaluation metric that uses pre-trained language models to estimate translation quality. It correlates better with human judgments than BLEU.

Low-Resource Translation

Low-resource languages pose unique challenges due to limited parallel data. LLMs offer several approaches to address this.

Transfer Learning Approaches

1. Zero-shot translation: Direct translation between language pairs not seen during training
2. Few-shot translation: Providing a small number of translation examples in the prompt
3. Cross-lingual transfer: Leveraging knowledge from high-resource languages

Pivot Translation

Pivot translation uses a high-resource language as an intermediate step, enabling translation between low-resource language pairs.

Practical Implementation

Translation with HuggingFace

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

# Load multilingual model
model_name = "facebook/mbart-large-50-many-to-many-mmt"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

# Translate French to German
text = "Bonjour, comment allez-vous aujourd'hui?"
tokenizer.src_lang = "fr_XX"
encoded = tokenizer(text, return_tensors="pt")
generated_tokens = model.generate(
    **encoded,
    forced_bos_token_id=tokenizer.lang_code_to_id["de_DE"]
)
translation = tokenizer.decode(generated_tokens[0], skip_special_tokens=True)
print(translation)  # "Hallo, wie geht es Ihnen heute?"

Translation with LLMs via Prompting

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "meta-llama/Llama-3-70B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")

prompt = """Translate the following English text to Japanese:
"The cherry blossoms in Tokyo are beautiful in spring."

Provide only the translation."""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
translation = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(translation)

Translation Challenges

Ambiguity Resolution

Translation ambiguity occurs when a source word or phrase has multiple possible translations. LLMs can resolve ambiguity using context.

Idiomatic Expressions

Idioms require cultural understanding beyond literal translation:

Cultural Adaptation

Translation often requires cultural adaptation to convey the same meaning effectively across cultures.

Evaluation Methodology

Human Evaluation

Human evaluation remains the gold standard for translation quality assessment:

1. Fluency: How natural does the translation read?
2. Adequacy: Does the translation convey the same meaning?
3. Terminology: Are technical terms translated correctly?
4. Style: Is the appropriate register maintained?

Automatic Metrics Comparison

Best Practices for Translation

Data Preparation

1. Parallel corpus cleaning: Remove misaligned sentence pairs
2. Deduplication: Remove duplicate translations
3. Domain balancing: Ensure representation across domains
4. Quality filtering: Use quality estimation to filter low-quality pairs

Model Selection

1. Resource availability: Choose models with sufficient language coverage
2. Domain specificity: Consider domain-adapted models
3. Latency requirements: Decoder-only LLMs may be slower than encoder-decoder
4. Cost constraints: Larger models offer better quality but higher inference costs

Practice Exercises

1. Evaluation: Compare BLEU and COMET scores for a set of translations. Which metric better captures translation quality for idiomatic expressions?

2. Implementation: Implement a zero-shot translation system using a multilingual LLM. Test translation between language pairs not explicitly represented in the training data.

3. Analysis: Analyze the translation quality of an LLM across different language families (e.g., Romance, Germanic, Slavic, Sino-Tibetan). What patterns emerge?

4. Research: Investigate the impact of prompt engineering on translation quality. How do different prompt formats affect translation accuracy?

What to Learn Next