GPT Family: Autoregressive Language Modeling
Module: Natural Language Processing | Difficulty: Advanced
Autoregressive Objective
Scaling Laws
In-Context Learning
Emergent Capabilities
| Capability | GPT-3 | GPT-4 | Emergence | |-----------|-------|-------|-----------| | Reasoning | Poor | Good | Yes | | Code | Poor | Good | Yes | | Math | Poor | Good | Yes |
Chain-of-Thought
where is the reasoning chain.
import torch
import torch.nn as nn
class GPTBlock(nn.Module):
def __init__(self, d_model, nhead, dropout=0.1):
super().__init__()
self.ln1 = nn.LayerNorm(d_model)
self.attn = nn.MultiheadAttention(d_model, nhead, batch_first=True)
self.ln2 = nn.LayerNorm(d_model)
self.ff = nn.Sequential(
nn.Linear(d_model, d_model*4), nn.GELU(), nn.Linear(d_model*4, d_model))
self.dropout = nn.Dropout(dropout)
def forward(self, x):
x_norm = self.ln1(x)
attn_mask = nn.Transformer.generate_square_subsequent_mask(x.size(1)).to(x.device)
attn_out, _ = self.attn(x_norm, x_norm, x_norm, attn_mask=attn_mask)
x = x + self.dropout(attn_out)
x = x + self.dropout(self.ff(self.ln2(x)))
return x
Research Insight: In-context learning is a emergent capability that arises at sufficient scale. The mechanism is not fully understood, but evidence suggests it implements a form of implicit gradient descent â the model learns to update its predictions based on provided examples.