RAG Deep Theory: Complete Architecture from First Principles

Retrieval-Augmented Generation (RAG) is the most important pattern in modern AI systems. It solves the fundamental limitation of LLMs — they only know what was in their training data. RAG lets AI access any information, at any time, without retraining.

This guide covers RAG from the mathematical foundations to production deployment.

Why RAG Exists

LLMs have three critical limitations:

1. Knowledge cutoff — GPT-4 doesn't know about events after its training date
2. Hallucination — LLMs generate plausible-sounding but false information
3. No private data access — LLMs don't know your company's internal documents

RAG solves all three by retrieving relevant information before generating answers.

Traditional LLM:     Question → LLM → Answer (from training data only)
RAG Pipeline:        Question → Retrieve → Context + Question → LLM → Answer (grounded in facts)

The Complete RAG Architecture

A RAG system has two distinct phases:

Phase 1: Indexing (Offline)

Documents are split, converted to vectors, and stored:

Documents → Chunking → Embedding → Vector Store

Phase 2: Querying (Online)

User questions are processed, relevant chunks retrieved, and answers generated:

Query → Embed → Search → Retrieve Top-K → Prompt LLM → Answer

Part 1: The Embedding Layer

What Are Embeddings?

An embedding is a dense vector representation of text in a high-dimensional space. Text with similar meanings are positioned close together in this space.

The Math Behind Embeddings

Embeddings use a neural network to map text to vectors. The key insight is contrastive learning — the model learns by comparing positive pairs (similar texts) against negative pairs (dissimilar texts).

Embedding Models Comparison

Choosing an Embedding Model

Part 2: The Chunking Layer

Why Chunking Matters

LLMs have limited context windows. Even GPT-4 has a 128K token limit. Your documents might be millions of tokens. More importantly, retrieval accuracy drops dramatically when you try to match a short query against a long document.

Chunking solves this by splitting documents into smaller, meaningful pieces.

Chunking Strategies

Strategy 1: Fixed-Size Chunking

Pros: Simple, predictable Cons: Can split mid-sentence, loses semantic boundaries

Strategy 2: Semantic Chunking

Pros: Respects document structure, better retrieval Cons: More complex, requires embedding model during indexing

Strategy 3: Recursive Character Splitting

Pros: Balances simplicity with quality, LangChain default Cons: Still not truly semantic

Strategy 4: Document-Structure Chunking

Optimal Chunk Size

Research shows the sweet spot depends on your use case:

Part 3: The Vector Store Layer

How Vector Search Works

Vector stores use approximate nearest neighbor (ANN) algorithms to find similar vectors efficiently. Searching every vector in a database would be too slow for millions of documents.

Indexing Algorithms

HNSW (Hierarchical Navigable Small World)

The most popular algorithm for production vector search:

Layer 3:  A ──────────────────── D
              \                 /
Layer 2:  A ──── B ──── C ──── D
            \   / \   / \   /
Layer 1:  A ── B ── C ── D ── E
          |    |    |    |    |
Data:     A    B    C    D    E

• Multi-layer graph structure
• Top layers = "highways" for fast long-distance navigation
• Bottom layers = fine-grained local search
• Search starts at top, drills down

IVF (Inverted File Index)

Documents → K-Means clustering → Inverted index per cluster
Query → Find nearest cluster → Search only within that cluster

Faster but less accurate than HNSW. Good for very large datasets.

Product Quantization (PQ)

Compresses vectors to reduce memory usage:

• 768-dimensional float32 vector = 3KB
• After PQ compression = ~96 bytes
• Trade-off: slight accuracy loss for massive memory savings

Popular Vector Databases

Choosing a Vector Database

Part 4: The Retrieval Layer

Basic Retrieval

Hybrid Search (Semantic + Keyword)

The best production systems combine semantic search with keyword matching:

Re-ranking

After initial retrieval, re-rank results with a more powerful model:

Query Expansion

Improve retrieval by expanding the query:

Part 5: The Generation Layer

Prompt Construction

The way you construct the prompt for the LLM dramatically affects answer quality:

Citation Generation

Production RAG systems must cite their sources:

Part 6: Advanced RAG Patterns

HyDE (Hypothetical Document Embeddings)

Instead of searching with the query, generate a hypothetical answer first, then search with that:

Self-RAG

Self-RAG adds reflection tokens to decide when to retrieve:

Corrective RAG (CRAG)

CRAG evaluates retrieval quality and takes different actions:

Graph RAG

Graph RAG uses knowledge graphs for multi-hop reasoning:

Part 7: Evaluation

RAG Evaluation Metrics

RAGAS Framework

The standard framework for RAG evaluation:

Part 8: Production Checklist

Summary

RAG is not just "search + generate." It's a complex system where every component — embedding model, chunking strategy, vector database, retrieval algorithm, prompt design, and generation model — must be carefully chosen and optimized together.

The key insight: RAG quality is determined by retrieval quality. The best LLM in the world cannot answer correctly if it receives the wrong context. Focus your optimization efforts on retrieval first.

Next Steps

1. Start with the basics: embeddings + ChromaDB
2. Add hybrid search for better retrieval
3. Implement re-ranking for precision
4. Build evaluation pipeline from day one
5. Graduate to advanced patterns (HyDE, Self-RAG) only after basics work well