Neural Scaling Laws: Chinchilla and Beyond

Module: Machine Learning | Difficulty: Advanced

Chinchilla Scaling

where = parameters, = tokens.

Compute-Optimal Training

Kaplan et al. (2020)

Empirical Results

| Model | Parameters | Tokens | Performance | |-------|-----------|--------|-------------| | GPT-3 | 175B | 300B | 86% | | Chinchilla | 70B | 1.4T | 90% | | LLaMA-2 | 70B | 2T | 92% |

import numpy as np

class ScalingLaw:
    def __init__(self):
        self.A, self.alpha = None, None
        self.B, self.beta = None, None
    def fit(self, N, D, losses):
        from scipy.optimize import curve_fit
        def loss_fn(x, A, alpha, B, beta, E):
            n, d = x
            return A/n**alpha + B/d**beta + E
        popt, _ = curve_fit(loss_fn, (N, D), losses, maxfev=10000)
        self.A, self.alpha, self.B, self.beta, self.E = popt
    def optimal_allocation(self, C):
        gamma = self.beta / (self.alpha + self.beta)
        N_opt = C ** (1 - gamma)
        D_opt = C ** gamma
        return N_opt, D_opt

Research Insight: Chinchilla showed that most language models are undertrained (too many parameters, too few tokens). The compute-optimal scaling suggests doubling parameters requires doubling data. This led to smaller, better-trained models like LLaMA.