RAG Evaluation and Benchmarking

Evaluating RAG systems requires assessing multiple components: the retriever, the generator, and their interaction. A system might retrieve perfect documents but generate hallucinations, or retrieve irrelevant documents but generate plausible-sounding answers.

Core Evaluation Metrics

Retrieval Metrics

Generation Metrics

Hallucination Detection

def detect_hallucinations(answer, context, llm):
    """Detect claims in the answer not supported by context."""
    prompt = f"""Analyze this answer for hallucinations.

Question: [hidden]
Context: {context}
Answer: {answer}

Identify each claim in the answer and mark it as:
- SUPPORTED: The claim is supported by the context
- UNSUPPORTED: The claim is not found in the context
- CONTRADICTED: The claim contradicts the context

Claims:"""
    
    response = llm.generate(prompt)
    return parse_claims(response)

RAGAS Framework

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)

def evaluate_rag_system(dataset, rag_pipeline):
    """Evaluate a RAG system using RAGAS metrics."""
    # Generate answers
    results = []
    for sample in dataset:
        answer = rag_pipeline.generate(sample["question"])
        results.append({
            "question": sample["question"],
            "answer": answer["response"],
            "contexts": answer["retrieved_contexts"],
            "ground_truth": sample["ground_truth"]
        })
    
    # Evaluate with RAGAS
    evaluation = evaluate(
        results,
        metrics=[
            faithfulness,
            answer_relevancy,
            context_precision,
            context_recall,
        ]
    )
    
    return evaluation

Evaluation Pipeline

class RAGEvaluationPipeline:
    def __init__(self, rag_system, llm_judge):
        self.rag_system = rag_system
        self.llm_judge = llm_judge
    
    def evaluate_single(self, question, ground_truth=None):
        """Evaluate a single query."""
        # Get RAG output
        output = self.rag_system.generate(question)
        
        # Evaluate faithfulness
        faithfulness = self.evaluate_faithfulness(
            output["response"],
            output["retrieved_contexts"]
        )
        
        # Evaluate answer relevance
        relevance = self.evaluate_answer_relevance(
            question,
            output["response"]
        )
        
        # Evaluate context precision
        precision = self.evaluate_context_precision(
            question,
            output["retrieved_contexts"],
            ground_truth
        )
        
        return {
            "question": question,
            "answer": output["response"],
            "faithfulness": faithfulness,
            "answer_relevance": relevance,
            "context_precision": precision,
            "num_retrieved": len(output["retrieved_contexts"])
        }
    
    def evaluate_faithfulness(self, answer, contexts):
        """Evaluate if answer is faithful to context."""
        prompt = f"""Rate how faithful this answer is to the provided context.

Context: {' '.join(contexts)}
Answer: {answer}

Rate from 0 to 1 (0 = completely hallucinated, 1 = fully supported):"""
        
        score = self.llm_judge.generate(prompt)
        return float(score)
    
    def evaluate_answer_relevance(self, question, answer):
        """Evaluate if answer addresses the question."""
        prompt = f"""Rate how relevant this answer is to the question.

Question: {question}
Answer: {answer}

Rate from 0 to 1 (0 = completely irrelevant, 1 = perfectly relevant):"""
        
        score = self.llm_judge.generate(prompt)
        return float(score)

Benchmarking RAG Systems

Benchmark Datasets

Evaluation Results Comparison

Practice Exercises

1. Metric Comparison: Compare faithfulness scores using different LLM judges (GPT-4 vs Claude vs Llama). How much do scores vary across judges?

2. Failure Analysis: Analyze 50 RAG failures. Categorize them as retrieval failures, generation failures, or integration failures. What is the most common failure mode?

3. A/B Testing: Design an A/B test to compare two RAG configurations. What metrics would you track and how long would you run the test?

4. Custom Metrics: Design a custom evaluation metric for RAG in a specific domain (e.g., medical, legal). What unique aspects would you measure?

Key Takeaways

What to Learn Next