Text Generation: Sampling, Decoding, and Control
Module: Natural Language Processing | Difficulty: Advanced
Temperature Sampling
Nucleus Sampling (Top-p)
Beam Search vs Sampling
| Method | Diversity | Coherence | Use Case |
|---|---|---|---|
| Greedy | Low | High | Translation |
| Beam | Medium | High | Summarization |
| Top-p | High | Medium | Creative |
| Top-k | High | Medium | Creative |
Controllable Generation
import torch
import torch.nn.functional as F
def nucleus_sample(logits, p=0.9, temperature=0.7):
logits = logits / temperature
sorted_logits, sorted_idx = torch.sort(logits, descending=True)
cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)
sorted_mask = cumulative_probs - F.softmax(sorted_logits, dim=-1) >= p
sorted_logits[sorted_mask] = float('-inf')
probs = F.softmax(sorted_logits, dim=-1)
return sorted_idx.gather(-1, torch.multinomial(probs, 1))
Research Insight: Nucleus sampling (top-p) produces more natural text than top-k because it adapts the number of candidates based on the probability distribution. When the model is confident, it samples from fewer candidates; when uncertain, it samples from more.