Autoregressive Models: Parallel Training and Compression

Module: Generative AI | Difficulty: Advanced

Autoregressive Decomposition

Causal Masking for Parallel Training

Compression Connection (Hutter, 2021)

Better predictor = better compressor = better model.

Speculative Decoding

1. Draft model generates tokens
2. Target model verifies in one pass
3. Accept where

Theorem: Output distribution is exactly the target model's distribution.

import torch
def speculative_decode(draft, target, prompt, gamma=4):
    tokens = prompt.clone()
    for _ in range(100//gamma):
        draft_t, draft_p = [], []
        x = tokens
        for _ in range(gamma):
            p = torch.softmax(draft(x)[:,-1], -1)
            t = torch.multinomial(p, 1)
            draft_t.append(t); draft_p.append(p)
            x = torch.cat([x, t], 1)
        tgt = target(torch.cat([tokens]+draft_t, 1))
        acc = 0
        for i, dt in enumerate(draft_t):
            tp = torch.softmax(tgt[:, tokens.size(1)+i-1], -1)
            if torch.rand(1) < min(1, tp[0,dt]/draft_p[i][0,dt]):
                acc += 1
            else: break
        tokens = torch.cat([tokens]+draft_t[:acc+1], 1)
    return tokens

| Context | Perplexity | MMLU | |---------|-----------|------| | 512 | 12.3 | 38.2 | | 4096 | 8.5 | 46.1 | | 32768 | 7.2 | 51.8 | | 131072 | 6.8 | 55.1 |