Neural Scaling Laws & Chinchilla

1. Power Law Scaling

Neural network performance follows power laws with respect to model size, dataset size, and compute:

where:

• : Loss (cross-entropy)
• : Number of parameters
• : Dataset size (tokens)
• : Compute budget (FLOPs)
• : Fitted constants

1.1 Scaling Law Dimensions

2. Scaling Law Log-Log Plots

<svg viewBox="0 0 800 400" xmlns="http://www.w3.org/2000/svg">
  <rect width="800" height="400" fill="#f8fafc"/>
  
  {/* Title */}
  <text x="400" y="30" text-anchor="middle" fill="#f8fafc" font-family="monospace" font-size="16" font-weight="bold">Neural Scaling Laws (Log-Log Scale)</text>
  
  {/* Axes */}
  <g transform="translate(80, 50)">
    {/* Y-axis (Loss) */}
    <line x1="0" y1="0" x2="0" y2="300" stroke="#64748b" stroke-width="2"/>
    <text x="-30" y="150" text-anchor="middle" fill="#94a3b8" font-family="monospace" font-size="10" transform="rotate(-90, -30, 150)">Loss (bits)</text>
    
    {/* X-axis (Compute) */}
    <line x1="0" y1="300" x2="640" y2="300" stroke="#64748b" stroke-width="2"/>
    <text x="320" y="330" text-anchor="middle" fill="#94a3b8" font-family="monospace" font-size="10">Compute (FLOPs, log₁₀)</text>
    
    {/* Y-axis ticks */}
    <text x="-10" y="30" text-anchor="end" fill="#64748b" font-family="monospace" font-size="8">3.5</text>
    <line x1="-5" y1="30" x2="0" y2="30" stroke="#64748b" stroke-width="1"/>
    
    <text x="-10" y="90" text-anchor="end" fill="#64748b" font-family="monospace" font-size="8">3.0</text>
    <line x1="-5" y1="90" x2="0" y2="90" stroke="#64748b" stroke-width="1"/>
    
    <text x="-10" y="150" text-anchor="end" fill="#64748b" font-family="monospace" font-size="8">2.5</text>
    <line x1="-5" y1="150" x2="0" y2="150" stroke="#64748b" stroke-width="1"/>
    
    <text x="-10" y="210" text-anchor="end" fill="#64748b" font-family="monospace" font-size="8">2.0</text>
    <line x1="-5" y1="210" x2="0" y2="210" stroke="#64748b" stroke-width="1"/>
    
    <text x="-10" y="270" text-anchor="end" fill="#64748b" font-family="monospace" font-size="8">1.5</text>
    <line x1="-5" y1="270" x2="0" y2="270" stroke="#64748b" stroke-width="1"/>
    
    {/* X-axis ticks */}
    <text x="0" y="315" text-anchor="middle" fill="#64748b" font-family="monospace" font-size="8">18</text>
    <text x="128" y="315" text-anchor="middle" fill="#64748b" font-family="monospace" font-size="8">20</text>
    <text x="256" y="315" text-anchor="middle" fill="#64748b" font-family="monospace" font-size="8">22</text>
    <text x="384" y="315" text-anchor="middle" fill="#64748b" font-family="monospace" font-size="8">24</text>
    <text x="512" y="315" text-anchor="middle" fill="#64748b" font-family="monospace" font-size="8">26</text>
    <text x="640" y="315" text-anchor="middle" fill="#64748b" font-family="monospace" font-size="8">28</text>
    
    {/* Scaling law line (power law: L = αC^-β) */}
    <path d="M 20 40 L 620 270" fill="none" stroke="#3b82f6" stroke-width="3"/>
    
    {/* Data points */}
    <circle cx="60" cy="80" r="5" fill="#22c55e"/>
    <circle cx="120" cy="110" r="5" fill="#22c55e"/>
    <circle cx="180" cy="135" r="5" fill="#22c55e"/>
    <circle cx="240" cy="160" r="5" fill="#22c55e"/>
    <circle cx="300" cy="185" r="5" fill="#22c55e"/>
    <circle cx="360" cy="210" r="5" fill="#22c55e"/>
    <circle cx="420" cy="230" r="5" fill="#22c55e"/>
    <circle cx="480" cy="250" r="5" fill="#22c55e"/>
    <circle cx="540" cy="265" r="5" fill="#22c55e"/>
    
    {/* Labels */}
    <text x="30" y="50" fill="#22c55e" font-family="monospace" font-size="9">GPT-3 175B</text>
    <text x="200" y="145" fill="#22c55e" font-family="monospace" font-size="9">PaLM 540B</text>
    <text x="400" y="220" fill="#22c55e" font-family="monospace" font-size="9">Chinchilla 70B</text>
    
    {/* Scaling law equation */}
    <g transform="translate(450, 50)">
      <rect width="180" height="60" rx="5" fill="#f8fafc" stroke="#334155"/>
      <text x="90" y="20" text-anchor="middle" fill="#60a5fa" font-family="monospace" font-size="10" font-weight="bold">Scaling Law</text>
      <text x="90" y="40" text-anchor="middle" fill="#f8fafc" font-family="monospace" font-size="9">L = α·C⁻β + ε</text>
      <text x="90" y="55" text-anchor="middle" fill="#22c55e" font-family="monospace" font-size="8">β ≈ 0.050</text>
    </g>
    
    {/* Power law annotation */}
    <text x="350" y="180" text-anchor="middle" fill="#f59e0b" font-family="monospace" font-size="10">Power Law: L ∝ C⁻⁰·⁰⁵</text>
  </g>
</svg>

3. Compute-Optimal Training

3.1 The Chinchilla Hypothesis

For a fixed compute budget , the optimal allocation between (parameters) and (tokens) follows:

with (linear scaling).

3.2 Chinchilla Optimal Ratios

Hoffmann et al. (2022) found:

The optimal tokens-per-parameter ratio:

3.3 Compute-Optimal Frontier

<svg viewBox="0 0 800 400" xmlns="http://www.w3.org/2000/svg">
  <rect width="800" height="400" fill="#f8fafc"/>
  
  {/* Title */}
  <text x="400" y="30" text-anchor="middle" fill="#f8fafc" font-family="monospace" font-size="16" font-weight="bold">Compute-Optimal Frontier (Chinchilla)</text>
  
  {/* Axes */}
  <g transform="translate(80, 50)">
    {/* Y-axis (Parameters) */}
    <line x1="0" y1="0" x2="0" y2="300" stroke="#64748b" stroke-width="2"/>
    <text x="-30" y="150" text-anchor="middle" fill="#94a3b8" font-family="monospace" font-size="10" transform="rotate(-90, -30, 150)">Parameters (log₁₀)</text>
    
    {/* X-axis (Tokens) */}
    <line x1="0" y1="300" x2="640" y2="300" stroke="#64748b" stroke-width="2"/>
    <text x="320" y="330" text-anchor="middle" fill="#94a3b8" font-family="monospace" font-size="10">Tokens (log₁₀)</text>
    
    {/* Compute-optimal frontier */}
    <path d="M 50 280 Q 200 200 350 150 Q 500 100 600 50" fill="none" stroke="#f59e0b" stroke-width="3"/>
    <text x="500" y="80" fill="#fbbf24" font-family="monospace" font-size="10">Compute-Optimal Frontier</text>
    
    {/* Under-trained region */}
    <path d="M 50 280 Q 100 250 150 200 Q 200 150 250 100" fill="none" stroke="#ec4899" stroke-width="2" stroke-dasharray="5,3"/>
    <text x="150" y="180" fill="#ec4899" font-family="monospace" font-size="9">Under-trained</text>
    
    {/* Over-trained region */}
    <path d="M 250 100 Q 350 150 450 200 Q 550 250 640 280" fill="none" stroke="#3b82f6" stroke-width="2" stroke-dasharray="5,3"/>
    <text x="450" y="180" fill="#60a5fa" font-family="monospace" font-size="9">Over-trained</text>
    
    {/* Model points */}
    <circle cx="100" cy="100" r="8" fill="#22c55e"/>
    <text x="100" y="85" text-anchor="middle" fill="#22c55e" font-family="monospace" font-size="8">GPT-3 175B</text>
    <text x="100" y="120" text-anchor="middle" fill="#94a3b8" font-family="monospace" font-size="7">300B tokens</text>
    
    <circle cx="200" cy="150" r="8" fill="#a855f7"/>
    <text x="200" y="135" text-anchor="middle" fill="#a855f7" font-family="monospace" font-size="8">PaLM 540B</text>
    <text x="200" y="170" text-anchor="middle" fill="#94a3b8" font-family="monospace" font-size="7">780B tokens</text>
    
    <circle cx="300" cy="120" r="8" fill="#22c55e"/>
    <text x="300" y="105" text-anchor="middle" fill="#22c55e" font-family="monospace" font-size="8">Chinchilla 70B</text>
    <text x="300" y="140" text-anchor="middle" fill="#94a3b8" font-family="monospace" font-size="7">1.4T tokens</text>
    
    <circle cx="400" cy="100" r="8" fill="#f59e0b"/>
    <text x="400" y="85" text-anchor="middle" fill="#f59e0b" font-family="monospace" font-size="8">LLaMA 65B</text>
    <text x="400" y="120" text-anchor="middle" fill="#94a3b8" font-family="monospace" font-size="7">1.4T tokens</text>
    
    {/* Legend */}
    <g transform="translate(500, 200)">
      <rect width="130" height="80" rx="5" fill="#f8fafc" stroke="#334155"/>
      <circle cx="20" cy="20" r="5" fill="#22c55e"/>
      <text x="35" y="24" fill="#94a3b8" font-family="monospace" font-size="8">Compute-optimal</text>
      <circle cx="20" cy="40" r="5" fill="#ec4899"/>
      <text x="35" y="44" fill="#94a3b8" font-family="monospace" font-size="8">Under-trained</text>
      <circle cx="20" cy="60" r="5" fill="#60a5fa"/>
      <text x="35" y="64" fill="#94a3b8" font-family="monospace" font-size="8">Over-trained</text>
    </g>
    
    {/* Grid */}
    <line x1="0" y1="60" x2="640" y2="60" stroke="#334155" stroke-width="0.5" stroke-dasharray="2,2"/>
    <line x1="0" y1="120" x2="640" y2="120" stroke="#334155" stroke-width="0.5" stroke-dasharray="2,2"/>
    <line x1="0" y1="180" x2="640" y2="180" stroke="#334155" stroke-width="0.5" stroke-dasharray="2,2"/>
    <line x1="0" y1="240" x2="640" y2="240" stroke="#334155" stroke-width="0.5" stroke-dasharray="2,2"/>
    
    <line x1="128" y1="0" x2="128" y2="300" stroke="#334155" stroke-width="0.5" stroke-dasharray="2,2"/>
    <line x1="256" y1="0" x2="256" y2="300" stroke="#334155" stroke-width="0.5" stroke-dasharray="2,2"/>
    <line x1="384" y1="0" x2="384" y2="300" stroke="#334155" stroke-width="0.5" stroke-dasharray="2,2"/>
    <line x1="512" y1="0" x2="512" y2="300" stroke="#334155" stroke-width="0.5" stroke-dasharray="2,2"/>
  </g>
</svg>

4. The Chinchilla Formula

4.1 Loss as a Function of N and D

The Chinchilla scaling law:

with fitted values:

• (irreducible loss)

4.2 Compute-Optimal Allocation

Given (approximate FLOPs), minimize subject to :

where:

• is a constant depending on

4.3 Practical Scaling Rules

5. Emergent Abilities

5.1 Definition

Emergent abilities (Wei et al., 2022) are abilities that:

1. Are not present in small models
2. Appear suddenly as model scale increases
3. Cannot be predicted by smooth extrapolation

5.2 Emergence Threshold

5.3 Examples of Emergence

5.4 Smooth Scaling Hypothesis

Alternative view (Schaeffer et al., 2023): Emergence may be an artifact of:

1. Non-linear metrics: Accuracy has a threshold
2. Evaluation granularity: Coarse evaluation misses gradual improvement

With smooth metrics (e.g., perplexity), scaling appears continuous:

6. Data Scaling Laws

6.1 Dataset Quality vs. Quantity

But quality matters:

• Curated data: Lower , higher effective
• Noisy data: Higher , lower effective

6.2 Data Mixing Laws

For multiple data sources with mixing proportions :

6.3 Repeat vs. Fresh Data

Epochwise scaling (Muennighoff et al., 2023):

where is the number of epochs (repeats).

Optimal epochs: for large datasets.

7. Compute Scaling Laws

7.1 FLOPs Estimation

For a Transformer with parameters and training tokens:

More precisely:

• Forward: FLOPs
• Backward: FLOPs

7.2 Wall-Clock Time

where throughput depends on:

• Hardware (GPU/TPU type)
• Parallelism strategy
• Communication overhead

7.3 Scaling Exponents

8. Beyond Chinchilla: Modern Scaling

8.1 Over-Training

Modern LLMs often over-train relative to Chinchilla:

Reason: Inference cost is dominated by model size, so over-training reduces total cost.

8.2 The Inference-Optimal Frontier

For many queries, inference-optimal has smaller and larger .

8.3 Multi-Modal Scaling

For vision-language models:

where subscripts and denote vision and language components.

9. Scaling Law Applications

9.1 Compute Budget Allocation

Given budget , predict optimal and :

def chinchilla_optimal(C, alpha=0.34, beta=0.28):
    """Compute Chinchilla-optimal N and D for budget C."""
    a = beta / (alpha + beta)
    b = alpha / (alpha + beta)
    
    # Proportionality constant
    G = 1.0  # Simplified; real value depends on A, B, alpha, beta
    
    N_opt = G * (C / 6) ** a
    D_opt = G ** (-1) * (C / 6) ** b
    
    return N_opt, D_opt

# Example: 10^23 FLOPs budget
C = 1e23
N, D = chinchilla_optimal(C)
print(f"Optimal: {N/1e9:.1f}B params, {D/1e9:.0f}B tokens")

9.2 Performance Prediction

9.3 Architecture Search

Scaling laws enable predicting the performance of new architectures without full training.

10. Limitations and Open Questions

1. Distribution shift: Scaling laws may change with new data distributions
2. Architecture dependence: Different architectures have different exponents
3. Task specificity: Scaling laws differ between pretraining and downstream tasks
4. Emergence prediction: Can we predict when new abilities will emerge?
5. Data quality: How does curation affect scaling exponents?

Neural scaling laws provide a powerful framework for understanding and predicting the behavior of large language models, enabling more efficient allocation of compute resources and guiding the development of next-generation AI systems.