The Geometry of Transformer Attention

Module: Generative AI | Difficulty: Advanced

Attention as Kernel Density Estimation

Theorem: Attention maps tokens to a convex hull of value vectors.

Implication: Deep transformers need residual connections because attention alone cannot expand the representational range.

Low-Rank Structure

When , attention acts as an information bottleneck.

RoPE: Rotary Position Encoding

Preserves:

import torch
def rope(x, pos):
    d = x.shape[-1]
    theta = 1.0 / (10000 ** (torch.arange(0, d, 2).float() / d))
    angles = pos.unsqueeze(-1) * theta.unsqueeze(0)
    return torch.cat([x[...,::2]*angles.cos() - x[...,1::2]*angles.sin(),
                       x[...,1::2]*angles.cos() + x[...,::2]*angles.sin()], -1)

Research Insight: Attention sinks (first token receiving disproportionate attention) exist because softmax must sum to 1, creating a "default" slot. Pruning this token degrades performance 5-15%.