BERT & GPT Family

1. Pre-Training Objectives

1.1 BERT: Masked Language Modelling (MLM)

BERT (Devlin et al., 2019) uses a bidirectional encoder trained with masked language modelling:

where is the set of masked positions (typically 15% of tokens).

Masking strategy:

• 80% of selected tokens: replace with [MASK]
• 10%: replace with random token
• 10%: keep original token

This mismatch between pre-training ([MASK] token) and fine-tuning (no [MASK]) is a limitation.

1.2 GPT: Autoregressive Language Modelling

GPT (Radford et al., 2018) trains a left-to-right decoder:

The model factorises the probability as:

1.3 T5: Span Corruption

T5 (Raffel et al., 2020) uses span corruption, masking contiguous spans:

The model learns to predict the spans given corrupted input.

1.4 Comparison of Objectives

2. Model Architectures

2.1 BERT Architecture

BERT uses a transformer encoder only:

BERT variants:

2.2 GPT Architecture

GPT uses a transformer decoder only (with causal masking):

GPT variants:

2.3 T5 Architecture

T5 uses a full encoder-decoder transformer:

Encoder: Bidirectional attention (no mask) Decoder: Causal attention + cross-attention to encoder

3. Scaling Laws

3.1 Kaplan Scaling Laws (OpenAI, 2020)

The loss of a neural language model scales as a power law with model size , dataset size , and compute budget :

Key insight: Larger models are more sample-efficient (better loss per token).

3.2 Chinchilla Scaling Laws (Hoffmann et al., 2022)

The Chinchilla paper showed that for optimal compute allocation:

The optimal ratio is approximately tokens per parameter.

Compute-optimal training:

3.3 Beyond Chinchilla

Recent work shows deviations from Chinchilla scaling:

1. Inference-optimal: Larger models with fewer training tokens may be better when inference cost matters
2. Over-training: Training on more tokens than Chinchilla-optimal improves downstream performance
3. Emergent abilities: Some capabilities only appear at certain scales

3.4 Scaling Law Predictions

4. Fine-Tuning Strategies

4.1 Standard Fine-Tuning

Update all parameters:

4.2 Parameter-Efficient Fine-Tuning (PEFT)

LoRA (Low-Rank Adaptation):

where (typically or ).

QLoRA: Quantised LoRA for even greater efficiency.

4.3 Instruction Tuning

Train on (instruction, response) pairs:

4.4 RLHF (Reinforcement Learning from Human Feedback)

Step 1: Supervised fine-tuning (SFT)

Step 2: Reward model training

Step 3: PPO optimisation

5. Perplexity and Evaluation

5.1 Perplexity

Perplexity is the exponential of the negative average log-likelihood:

Interpretation: PPL is the effective vocabulary size the model is "confused" between at each step.

5.2 Bits Per Character/Token

5.3 Evaluation Benchmarks

6. Code Examples

6.1 BERT Fine-Tuning

import torch
import torch.nn as nn
from transformers import BertModel, BertTokenizer

class BertForClassification(nn.Module):
    """BERT fine-tuning for text classification."""
    
    def __init__(self, model_name, num_classes, dropout=0.1):
        super().__init__()
        self.bert = BertModel.from_pretrained(model_name)
        self.dropout = nn.Dropout(dropout)
        self.classifier = nn.Linear(self.bert.config.hidden_size, num_classes)
    
    def forward(self, input_ids, attention_mask=None, token_type_ids=None):
        """
        Forward pass.
        
        Parameters
        ----------
        input_ids : Tensor (batch, seq_len)
        attention_mask : Tensor (batch, seq_len)
        token_type_ids : Tensor (batch, seq_len)
        
        Returns
        -------
        logits : Tensor (batch, num_classes)
        """
        outputs = self.bert(
            input_ids=input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids
        )
        
        # Use [CLS] token representation
        pooled = outputs.pooler_output  # (batch, hidden)
        pooled = self.dropout(pooled)
        logits = self.classifier(pooled)
        
        return logits
    
    def freeze_bert(self, num_layers_to_freeze=None):
        """Freeze BERT layers for efficient fine-tuning."""
        if num_layers_to_freeze is None:
            # Freeze all but classifier
            for param in self.bert.parameters():
                param.requires_grad = False
        else:
            # Freeze embeddings and first N layers
            for param in self.bert.embeddings.parameters():
                param.requires_grad = False
            for i in range(num_layers_to_freeze):
                for param in self.bert.encoder.layer[i].parameters():
                    param.requires_grad = False
        
        # Count trainable parameters
        trainable = sum(p.numel() for p in self.parameters() if p.requires_grad)
        total = sum(p.numel() for p in self.parameters())
        print(f"Trainable: {trainable:,} / {total:,} ({100*trainable/total:.1f}%)")


# Example: Fine-tune BERT
model = BertForClassification('bert-base-uncased', num_classes=2)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Freeze all but last 2 BERT layers
model.freeze_bert(num_layers_to_freeze=10)

# Forward pass
text = ["This movie is great!", "This movie is terrible!"]
inputs = tokenizer(text, return_tensors='pt', padding=True, truncation=True)
logits = model(**inputs)

print(f"Logits shape: {logits.shape}")
print(f"Predictions: {logits.argmax(dim=-1)}")

6.2 LoRA Implementation

import torch
import torch.nn as nn
import math

class LoRALayer(nn.Module):
    """
    Low-Rank Adaptation (LoRA) layer.
    
    ΔW = B × A, where B ∈ R^{d×r}, A ∈ R^{r×d}
    """
    
    def __init__(self, in_features, out_features, rank=8, alpha=16):
        super().__init__()
        self.rank = rank
        self.alpha = alpha
        self.scaling = alpha / rank
        
        # Low-rank matrices
        self.A = nn.Parameter(torch.randn(rank, in_features) / math.sqrt(rank))
        self.B = nn.Parameter(torch.zeros(out_features, rank))
        
        # Original weight (frozen)
        self.weight = nn.Parameter(torch.randn(out_features, in_features))
        self.weight.requires_grad = False
        
        # Optional bias
        self.bias = nn.Parameter(torch.zeros(out_features))
    
    def forward(self, x):
        # Original linear transformation
        out = x @ self.weight.T + self.bias
        
        # LoRA adaptation
        lora = x @ self.A.T @ self.B.T * self.scaling
        
        return out + lora
    
    def merge_weights(self):
        """Merge LoRA weights into original weight for inference."""
        with torch.no_grad():
            self.weight += self.scaling * self.B @ self.A
            self.A.requires_grad = False
            self.B.requires_grad = False


class LoRAModel(nn.Module):
    """Model with LoRA applied to specified layers."""
    
    def __init__(self, base_model, target_modules=['q_proj', 'v_proj'], rank=8):
        super().__init__()
        self.base_model = base_model
        self.lora_layers = nn.ModuleDict()
        
        # Apply LoRA to target modules
        for name, module in base_model.named_modules():
            if any(target in name for target in target_modules):
                if isinstance(module, nn.Linear):
                    lora = LoRALayer(
                        module.in_features,
                        module.out_features,
                        rank=rank
                    )
                    self.lora_layers[name.replace('.', '_')] = lora
        
        print(f"Applied LoRA to {len(self.lora_layers)} layers")
    
    def forward(self, x):
        return self.base_model(x)


# Example: Apply LoRA to a model
class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.q_proj = nn.Linear(512, 512)
        self.v_proj = nn.Linear(512, 512)
        self.out = nn.Linear(512, 10)
    
    def forward(self, x):
        q = self.q_proj(x)
        v = self.v_proj(x)
        return self.out(q + v)

base_model = SimpleModel()
lora_model = LoRAModel(base_model, target_modules=['q_proj', 'v_proj'], rank=8)

# Count parameters
total = sum(p.numel() for p in base_model.parameters())
lora_params = sum(p.numel() for p in lora_model.lora_layers.parameters())
print(f"Base model: {total:,} params")
print(f"LoRA params: {lora_params:,} params ({100*lora_params/total:.2f}%)")

6.3 Scaling Law Computation

import numpy as np
import matplotlib.pyplot as plt

class ScalingLawPredictor:
    """
    Neural scaling law predictions.
    
    Based on Chinchilla (Hoffmann et al., 2022).
    """
    
    def __init__(self):
        # Chinchilla parameters (from paper)
        self.alpha_N = 0.34
        self.alpha_D = 0.28
        self.E = 1.46  # irreducible loss
        self.A = 4.06e8
        self.B = 1.69e9
    
    def compute_loss(self, N, D):
        """
        Compute predicted loss for given model size and dataset size.
        
        Parameters
        ----------
        N : int or array
            Number of parameters
        D : int or array
            Number of training tokens
        
        Returns
        -------
        L : float or array
            Predicted loss (cross-entropy)
        """
        return self.A / (N ** self.alpha_N) + self.B / (D ** self.alpha_D) + self.E
    
    def optimal_allocation(self, C):
        """
        Compute optimal model size and dataset size for given compute budget.
        
        Parameters
        ----------
        C : float
            Compute budget in FLOPs
        
        Returns
        -------
        N_opt, D_opt : float
            Optimal model size and dataset size
        """
        # From Chinchilla: C ≈ 6ND
        # Optimal: N ∝ C^0.5, D ∝ C^0.5
        N_opt = (C / 6) ** 0.5
        D_opt = (C / 6) ** 0.5
        return N_opt, D_opt
    
    def predict_from_compute(self, C):
        """Predict loss from compute budget."""
        N_opt, D_opt = self.optimal_allocation(C)
        return self.compute_loss(N_opt, D_opt)
    
    def plot_scaling_laws(self):
        """Visualise scaling laws."""
        fig, axes = plt.subplots(1, 3, figsize=(15, 4))
        
        # Loss vs Model Size
        N_range = np.logspace(7, 11, 100)
        D_fixed = 1e11
        L_N = self.compute_loss(N_range, D_fixed)
        
        axes[0].loglog(N_range, L_N)
        axes[0].set_xlabel('Model Size (N)')
        axes[0].set_ylabel('Loss')
        axes[0].set_title(f'Loss vs Model Size (D={D_fixed:.0e})')
        axes[0].grid(True)
        
        # Loss vs Dataset Size
        D_range = np.logspace(9, 13, 100)
        N_fixed = 1e9
        L_D = self.compute_loss(N_fixed, D_range)
        
        axes[1].loglog(D_range, L_D)
        axes[1].set_xlabel('Dataset Size (D)')
        axes[1].set_ylabel('Loss')
        axes[1].set_title(f'Loss vs Dataset Size (N={N_fixed:.0e})')
        axes[1].grid(True)
        
        # Loss vs Compute
        C_range = np.logspace(17, 25, 100)
        L_C = self.predict_from_compute(C_range)
        
        axes[2].loglog(C_range, L_C)
        axes[2].set_xlabel('Compute (FLOPs)')
        axes[2].set_ylabel('Loss')
        axes[2].set_title('Loss vs Compute (Optimal Allocation)')
        axes[2].grid(True)
        
        plt.tight_layout()
        return fig
    
    def compare_models(self):
        """Compare predictions with actual model performance."""
        models = {
            'GPT-2 Small': {'N': 117e6, 'D': 1e10, 'actual_loss': 3.29},
            'GPT-2 Medium': {'N': 345e6, 'D': 1e10, 'actual_loss': 2.96},
            'GPT-2 XL': {'N': 1.5e9, 'D': 1e10, 'actual_loss': 2.80},
            'GPT-3': {'N': 175e9, 'D': 300e9, 'actual_loss': 2.83},
            'Chinchilla': {'N': 70e9, 'D': 1.4e12, 'actual_loss': 2.67},
            'LLaMA-2 70B': {'N': 70e9, 'D': 2e12, 'actual_loss': 2.55},
        }
        
        print(f"{'Model':<20} {'N':<12} {'D':<12} {'Predicted':<10} {'Actual':<10} {'Error'}")
        print("-" * 75)
        
        for name, data in models.items():
            predicted = self.compute_loss(data['N'], data['D'])
            error = abs(predicted - data['actual_loss']) / data['actual_loss'] * 100
            print(f"{name:<20} {data['N']:<12.1e} {data['D']:<12.1e} "
                  f"{predicted:<10.3f} {data['actual_loss']:<10.3f} {error:.1f}%")


# Example: Scaling law analysis
predictor = ScalingLawPredictor()

# Compare with actual models
predictor.compare_models()

# Optimal allocation for different compute budgets
print("\nOptimal allocation:")
for C in [1e18, 1e20, 1e22, 1e24]:
    N_opt, D_opt = predictor.optimal_allocation(C)
    L = predictor.predict_from_compute(C)
    print(f"C={C:.0e}: N={N_opt:.2e}, D={D_opt:.2e}, Loss={L:.3f}")

7. Summary

1. BERT uses masked language modelling for bidirectional pre-training, achieving strong NLU performance.

2. GPT uses autoregressive language modelling, scaling to large language models with emergent capabilities.

3. Scaling laws predict performance from model size, dataset size, and compute, with Chinchilla providing optimal allocation.

4. Fine-tuning methods range from full fine-tuning to parameter-efficient approaches like LoRA.

5. RLHF aligns language models with human preferences through reward modelling and policy optimisation.

References

• Devlin, J., et al. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL.
• Radford, A., et al. (2018). Improving language understanding by generative pre-training. OpenAI.
• Raffel, C., et al. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR.
• Kaplan, J., et al. (2020). Scaling laws for neural language models. arXiv.
• Hoffmann, J., et al. (2022). Training compute-optimal large language models. NeurIPS.
• Hu, E., et al. (2022). LoRA: Low-rank adaptation of large language models. ICLR.
• Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS.