RAG: Retrieval-Augmented Generation
Module: Natural Language Processing | Difficulty: Advanced
RAG Framework
Dense Passage Retrieval (DPR)
Retrieval + Generation
- Encode query
- Retrieve top-k passages
- Concatenate with query
- Generate answer
Metrics
| Metric | Retrieval | Generation | End-to-End |
|---|---|---|---|
| Recall@10 | 85% | - | - |
| Exact Match | - | 65% | 60% |
| F1 | - | 72% | 68% |
import torch
import torch.nn as nn
import torch.nn.functional as F
class RAGModel(nn.Module):
def __init__(self, query_encoder, passage_encoder, generator, top_k=5):
super().__init__()
self.q_enc = query_encoder
self.p_enc = passage_encoder
self.generator = generator
self.top_k = top_k
def retrieve(self, query_ids, passage_ids):
q_emb = F.normalize(self.q_enc(query_ids), dim=1)
p_emb = F.normalize(self.p_enc(passage_ids), dim=1)
scores = q_emb @ p_emb.T
top_k = scores.topk(self.top_k, dim=1)
return top_k.indices
def forward(self, query_ids, passage_ids, answer_ids=None):
retrieved = self.retrieve(query_ids, passage_ids)
if answer_ids is not None:
return self.generator(passage_ids[retrieved], answer_ids)
return self.generator.generate(passage_ids[retrieved])
Research Insight: RAG combines the strengths of parametric knowledge (stored in model weights) with non-parametric knowledge (stored in the retrieval corpus). This enables up-to-date knowledge without retraining, but requires careful balance between retrieval quality and generation fluency.