Language Model Alignment: RLHF and Constitutional AI

Module: Natural Language Processing | Difficulty: Advanced

RLHF Pipeline

1. Supervised fine-tuning
2. Reward model training
3. PPO optimization

Reward Model

PPO Objective

Constitutional AI

import torch
import torch.nn as nn

class RewardModel(nn.Module):
    def __init__(self, bert_model):
        super().__init__()
        self.bert = bert_model
        self.reward_head = nn.Linear(768, 1)
    def forward(self, input_ids, attention_mask):
        outputs = self.bert(input_ids, attention_mask=attention_mask)
        cls_output = outputs.last_hidden_state[:, 0]
        return self.reward_head(cls_output).squeeze(-1)

Research Insight: RLHF reduces harmful outputs by 60-80% but can also reduce helpfulness by 10-20%. Constitutional AI attempts to automate the alignment process by having the model critique its own outputs against a set of principles.