Agent Evaluation Framework
What is an Agent Evaluation Framework?
Agent evaluation frameworks systematically measure agent performance across multiple dimensions: accuracy, latency, cost, reliability, and safety. They enable data-driven improvements by providing consistent, reproducible benchmarks.
The key components are: test case management, execution tracing, metric calculation, comparative analysis, and reporting. Without evaluation, agent improvements are guesswork; with it, every change is measurable.
Why This Matters
LLM behavior is non-deterministic â the same prompt can produce different outputs. Without systematic evaluation, you cannot determine if a change actually improved performance or just happened to work on a few examples. Evaluation frameworks transform subjective opinions into objective metrics.
Real-World Analogy
An agent evaluation framework is like a car crash test facility. Before a car goes to production, it undergoes standardized tests (crash tests, emissions, performance). Similarly, before an agent goes to production, it needs standardized tests across accuracy, safety, latency, and cost. You wouldn't buy a car that wasn't crash-tested, and you shouldn't deploy an agent that wasn't evaluation-tested.
Project Overview
We will build an agent evaluation framework that:
- Manages test cases with expected outputs and metadata
- Traces agent execution with detailed step logging
- Calculates accuracy, latency, and cost metrics
- Compares different agent configurations statistically
- Generates comprehensive evaluation reports
- Provides observability through structured traces
Expected outcome: A framework that enables systematic, reproducible agent improvement.
Difficulty: Advanced (requires understanding of testing methodologies, metrics, and observability)
Tools & Setup
| Tool | Version | Purpose |
|---|---|---|
| Python | 3.11+ | Core language |
| pytest | 7.0+ | Test execution |
| openai | 1.0+ | LLM backbone |
| pandas | 2.0+ | Metrics analysis |
| numpy | 1.24+ | Statistical analysis |
Step 1: Environment Setup
python -m venv venv
source venv/bin/activate
pip install pytest openai pandas numpy
export OPENAI_API_KEY="sk-your-key"
Step 2: Test Case Manager
# test_cases/manager.py
from pydantic import BaseModel, Field
from typing import List, Dict, Optional
import json
import logging
import hashlib
logger = logging.getLogger(__name__)
class TestCase(BaseModel):
id: str
input: str
expected_output: Optional[str] = None
expected_tool: Optional[str] = None
tags: List[str] = Field(default_factory=list)
difficulty: str = "medium"
category: str = "general"
metadata: Dict = Field(default_factory=dict)
class TestCaseManager:
"""Manage test cases for agent evaluation with filtering and versioning."""
def __init__(self):
self.test_cases: List[TestCase] = []
self._hash_cache: Dict[str, str] = {}
def add_test_case(self, test_case: TestCase) -> None:
existing_ids = {tc.id for tc in self.test_cases}
if test_case.id in existing_ids:
logger.warning("Test case %s already exists, skipping", test_case.id)
return
self.test_cases.append(test_case)
self._hash_cache[test_case.id] = hashlib.md5(
test_case.model_dump_json().encode()
).hexdigest()
def load_from_file(self, file_path: str) -> None:
try:
with open(file_path, "r") as f:
data = json.load(f)
for item in data:
self.test_cases.append(TestCase(**item))
logger.info("Loaded %d test cases from %s", len(data), file_path)
except Exception as e:
logger.error("Failed to load test cases: %s", e)
def save_to_file(self, file_path: str) -> None:
data = [tc.model_dump() for tc in self.test_cases]
with open(file_path, "w") as f:
json.dump(data, f, indent=2)
def get_by_tag(self, tag: str) -> List[TestCase]:
return [tc for tc in self.test_cases if tag in tc.tags]
def get_by_difficulty(self, difficulty: str) -> List[TestCase]:
return [tc for tc in self.test_cases if tc.difficulty == difficulty]
def get_by_category(self, category: str) -> List[TestCase]:
return [tc for tc in self.test_cases if tc.category == category]
def create_test_suite(
self,
tags: Optional[List[str]] = None,
difficulty: Optional[str] = None,
category: Optional[str] = None,
max_tests: Optional[int] = None,
) -> List[TestCase]:
suite = self.test_cases.copy()
if tags:
suite = [tc for tc in suite if any(t in tc.tags for t in tags)]
if difficulty:
suite = [tc for tc in suite if tc.difficulty == difficulty]
if category:
suite = [tc for tc in suite if tc.category == category]
if max_tests:
suite = suite[:max_tests]
return suite
def get_statistics(self) -> Dict:
difficulties = {}
categories = {}
for tc in self.test_cases:
difficulties[tc.difficulty] = difficulties.get(tc.difficulty, 0) + 1
categories[tc.category] = categories.get(tc.category, 0) + 1
return {
"total": len(self.test_cases),
"by_difficulty": difficulties,
"by_category": categories,
}
Step 3: Execution Tracer
# tracing/tracer.py
from typing import Dict, List, Any, Optional
import time
from dataclasses import dataclass, field
from datetime import datetime
import logging
import uuid
logger = logging.getLogger(__name__)
@dataclass
class TraceStep:
step_id: int
name: str
input_data: Any
output_data: Any
duration_ms: float
tokens_used: int = 0
cost: float = 0.0
metadata: Dict = field(default_factory=dict)
timestamp: str = field(default_factory=lambda: datetime.now().isoformat())
class ExecutionTracer:
"""Trace agent execution for debugging and evaluation."""
def __init__(self):
self.traces: List[Dict] = []
self.current_trace: Optional[Dict] = None
self.steps: List[TraceStep] = []
def start_trace(self, test_case_id: str, input_data: str) -> None:
trace_id = f"trace_{uuid.uuid4().hex[:12]}"
self.current_trace = {
"id": trace_id,
"test_case_id": test_case_id,
"input": input_data,
"start_time": time.time(),
"start_timestamp": datetime.now().isoformat(),
"steps": [],
"status": "running",
}
self.steps = []
logger.info("Started trace %s for test case %s", trace_id, test_case_id)
def add_step(
self,
name: str,
input_data: Any,
output_data: Any,
duration_ms: float,
tokens_used: int = 0,
cost: float = 0.0,
metadata: Optional[Dict] = None,
) -> None:
step = TraceStep(
step_id=len(self.steps) + 1,
name=name,
input_data=input_data,
output_data=output_data,
duration_ms=duration_ms,
tokens_used=tokens_used,
cost=cost,
metadata=metadata or {},
)
self.steps.append(step)
def end_trace(self, output: str, success: bool = True) -> None:
if not self.current_trace:
return
self.current_trace["output"] = output
self.current_trace["success"] = success
self.current_trace["end_time"] = time.time()
self.current_trace["end_timestamp"] = datetime.now().isoformat()
self.current_trace["duration_ms"] = (
self.current_trace["end_time"] - self.current_trace["start_time"]
) * 1000
self.current_trace["steps"] = [
{
"step_id": s.step_id,
"name": s.name,
"input": str(s.input_data)[:200],
"output": str(s.output_data)[:200],
"duration_ms": s.duration_ms,
"tokens_used": s.tokens_used,
"cost": s.cost,
"timestamp": s.timestamp,
}
for s in self.steps
]
self.current_trace["total_tokens"] = sum(s.tokens_used for s in self.steps)
self.current_trace["total_cost"] = sum(s.cost for s in self.steps)
self.current_trace["step_count"] = len(self.steps)
self.traces.append(self.current_trace)
logger.info(
"Trace %s completed: success=%s, duration=%.1fms, tokens=%d",
self.current_trace["id"],
success,
self.current_trace["duration_ms"],
self.current_trace["total_tokens"],
)
self.current_trace = None
self.steps = []
def get_traces(self) -> List[Dict]:
return self.traces
def get_failed_traces(self) -> List[Dict]:
return [t for t in self.traces if not t.get("success")]
def get_summary(self) -> Dict:
if not self.traces:
return {"total": 0}
total = len(self.traces)
successes = sum(1 for t in self.traces if t.get("success"))
return {
"total_traces": total,
"successes": successes,
"failures": total - successes,
"success_rate": round(successes / total * 100, 2),
}
Step 4: Metrics Calculator and Framework
# metrics/calculator.py
from typing import Dict, List, Optional
import numpy as np
import logging
logger = logging.getLogger(__name__)
class MetricsCalculator:
"""Calculate evaluation metrics for agent traces with statistical analysis."""
def calculate_accuracy(
self, predicted: str, expected: str, method: str = "semantic"
) -> float:
if not expected:
return 1.0
if method == "exact":
return 1.0 if predicted.strip() == expected.strip() else 0.0
elif method == "contains":
return 1.0 if expected.lower() in predicted.lower() else 0.0
elif method == "semantic":
return self._semantic_similarity(predicted, expected)
elif method == "fuzzy":
return self._fuzzy_match(predicted, expected)
return 0.0
def _semantic_similarity(self, text1: str, text2: str) -> float:
words1 = set(text1.lower().split())
words2 = set(text2.lower().split())
intersection = words1 & words2
union = words1 | words2
return len(intersection) / len(union) if union else 0.0
def _fuzzy_match(self, text1: str, text2: str) -> float:
text1_lower = text1.lower().strip()
text2_lower = text2.lower().strip()
if text1_lower == text2_lower:
return 1.0
if text2_lower in text1_lower:
return 0.9
words1 = set(text1_lower.split())
words2 = set(text2_lower.split())
if not words2:
return 0.0
return len(words1 & words2) / len(words2)
def calculate_metrics(self, traces: List[Dict]) -> Dict:
if not traces:
return {"total": 0}
total = len(traces)
successes = sum(1 for t in traces if t.get("success"))
durations = [t.get("duration_ms", 0) for t in traces]
tokens = [t.get("total_tokens", 0) for t in traces]
costs = [t.get("total_cost", 0) for t in traces]
return {
"total_tests": total,
"successes": successes,
"failures": total - successes,
"success_rate": round(successes / total * 100, 2),
"avg_duration_ms": round(np.mean(durations), 2) if durations else 0,
"p50_duration": round(np.percentile(durations, 50), 2) if durations else 0,
"p95_duration": round(np.percentile(durations, 95), 2) if durations else 0,
"p99_duration": round(np.percentile(durations, 99), 2) if durations else 0,
"avg_tokens": round(np.mean(tokens), 2) if tokens else 0,
"total_cost": round(sum(costs), 4),
"avg_cost": round(np.mean(costs), 4) if costs else 0,
}
def compare_results(self, results1: Dict, results2: Dict) -> Dict:
comparison: Dict = {}
for key in results1:
if key in results2 and isinstance(results1[key], (int, float)):
baseline = results1[key]
comparison_val = results2[key]
diff = comparison_val - baseline
pct = (diff / baseline * 100) if baseline != 0 else 0
comparison[key] = {
"baseline": baseline,
"comparison": comparison_val,
"diff": round(diff, 4),
"pct_change": round(pct, 2),
}
return comparison
# framework.py
from test_cases.manager import TestCaseManager, TestCase
from tracing.tracer import ExecutionTracer
from metrics.calculator import MetricsCalculator
from typing import Dict, List, Callable, Any
import time
import logging
logger = logging.getLogger(__name__)
class AgentEvaluationFramework:
"""Complete evaluation framework for agent testing and benchmarking."""
def __init__(self):
self.test_manager = TestCaseManager()
self.tracer = ExecutionTracer()
self.metrics = MetricsCalculator()
self.results: List[Dict] = []
async def evaluate_agent(
self,
agent_func: Callable,
test_cases: Optional[List[TestCase]] = None,
accuracy_threshold: float = 0.5,
) -> Dict:
if test_cases is None:
test_cases = self.test_manager.test_cases
logger.info("Starting evaluation with %d test cases", len(test_cases))
for tc in test_cases:
self.tracer.start_trace(tc.id, tc.input)
start = time.time()
try:
output = await agent_func(tc.input)
duration = (time.time() - start) * 1000
self.tracer.add_step("agent_execution", tc.input, output, duration)
accuracy = self.metrics.calculate_accuracy(
output, tc.expected_output or ""
)
self.tracer.end_trace(output, success=accuracy > accuracy_threshold)
self.results.append({
"test_case_id": tc.id,
"input": tc.input,
"expected": tc.expected_output,
"actual": output,
"accuracy": accuracy,
"success": accuracy > accuracy_threshold,
"duration_ms": duration,
})
except Exception as e:
self.tracer.end_trace(str(e), success=False)
self.results.append({
"test_case_id": tc.id,
"input": tc.input,
"expected": tc.expected_output,
"actual": str(e),
"accuracy": 0.0,
"success": False,
})
return self.metrics.calculate_metrics(self.tracer.get_traces())
def generate_report(self) -> str:
metrics = self.metrics.calculate_metrics(self.tracer.get_traces())
report_lines = [
"# Agent Evaluation Report",
"",
"## Summary",
f"- Total Tests: {metrics['total_tests']}",
f"- Success Rate: {metrics['success_rate']:.1f}%",
f"- Avg Duration: {metrics['avg_duration_ms']:.0f}ms",
f"- P95 Duration: {metrics['p95_duration']:.0f}ms",
f"- Avg Tokens: {metrics['avg_tokens']:.0f}",
f"- Total Cost: ${metrics['total_cost']:.4f}",
"",
"## Failed Tests",
]
failed = [r for r in self.results if not r["success"]]
for r in failed:
expected = str(r["expected"])[:50] if r["expected"] else "N/A"
actual = str(r["actual"])[:50]
report_lines.append(f"- {r['test_case_id']}: Expected '{expected}' got '{actual}'")
if not failed:
report_lines.append("- No failed tests!")
return "\n".join(report_lines)
Why This Matters
Without evaluation, you're flying blind. You might change a prompt and see better results on 3 test cases, but miss that performance degraded on 50 others. Evaluation frameworks provide the statistical rigor needed to make confident improvements.
Real-World Analogy
An evaluation framework is like a clinical trial for drugs. Before a drug is approved, it undergoes rigorous testing with control groups, statistical analysis, and multiple trials. Similarly, before an agent change is deployed, it should be tested against a benchmark suite with statistical significance testing.
Mathematical Foundation
F1 Score:
Where:
- (correct positive predictions / total positive predictions)
- (correct positive predictions / total actual positives)
Intuition: Harmonic mean of precision and recall. F1 balances false positives and false negatives, ideal for imbalanced evaluation datasets.
Cost per Successful Task:
Intuition: Measures cost efficiency. Lower is better. An agent with 0.05C_{\text{eff}} = <MathBlock tex=0.10 />.
Performance Considerations
| Metric | Value | Notes |
|---|---|---|
| Test Execution | 1-10s | Per test case (LLM dependent) |
| Metrics Calculation | <100ms | For 100 traces |
| Report Generation | 1-2s | Full report |
| Trace Upload | 100-500ms | Per trace |
| Benchmark Suite | 5-30min | 100+ test cases |
| Statistical Significance | 30+ samples | For reliable comparisons |
Security Considerations
- Test Data Privacy: Never use real customer data in test cases; use synthetic data
- API Key Protection: Use environment variables for LLM API keys in evaluation
- Result Integrity: Store evaluation results in tamper-proof storage for audit
- Access Control: Restrict access to evaluation results and agent configurations
- Cost Limits: Set budget limits for evaluation runs to prevent unexpected charges
Testing & Evaluation
import pytest
from framework import AgentEvaluationFramework
from test_cases.manager import TestCase
@pytest.mark.asyncio
async def test_framework():
framework = AgentEvaluationFramework()
framework.test_manager.add_test_case(TestCase(
id="test1", input="hello", expected_output="hi there"
))
async def mock_agent(x):
return "hi there"
results = await framework.evaluate_agent(mock_agent)
assert results["success_rate"] == 100.0
def test_metrics_calculator():
calc = MetricsCalculator()
score = calc.calculate_accuracy("The answer is 42", "42", method="contains")
assert score == 1.0
def test_test_suite_creation():
manager = TestCaseManager()
manager.add_test_case(TestCase(id="t1", input="q1", difficulty="easy", tags=["basic"]))
manager.add_test_case(TestCase(id="t2", input="q2", difficulty="hard", tags=["advanced"]))
suite = manager.create_test_suite(difficulty="easy")
assert len(suite) == 1
Interview Q&A
Q1: Why is systematic evaluation important for LLM agents? A: LLM behavior is non-deterministic â the same prompt can produce different outputs. Systematic evaluation provides reproducible metrics to measure improvement, detect regressions, and make data-driven decisions about model changes. It transforms subjective opinions into objective measurements.
Q2: What is the difference between precision and recall in agent evaluation? A: Precision measures how many agent outputs are correct among all outputs generated. Recall measures how many correct outputs the agent generates among all possible correct answers. High precision means few false positives; high recall means few false negatives.
Q3: How would you handle non-deterministic agent outputs in evaluation? A: Run each test case 3-5 times and average results, use semantic similarity instead of exact matching, implement deterministic decoding (temperature=0), and maintain a baseline for statistical comparison. Use confidence intervals to quantify uncertainty.
Q4: What metrics matter most for production agents? A: Success rate (task completion), latency (P50 and P95), cost per task, error rate, and safety violation rate. Track all metrics over time to detect degradation. Include both technical metrics (latency, tokens) and business metrics (task completion, user satisfaction).
Q5: How does observability improve evaluation? A: Execution traces provide visibility into agent behavior and failure modes. They enable debugging why specific test cases failed, identifying bottlenecks, and understanding decision patterns. Structured traces also enable comparing different agent configurations.
Q6: What is the difference between unit tests and integration tests for agents? A: Unit tests verify individual components (tool execution, prompt parsing). Integration tests verify end-to-end agent behavior (full task completion). Both are needed â unit tests for speed and isolation, integration tests for correctness and real-world behavior.
Q7: How would you benchmark agent improvements? A: Maintain a frozen test suite, run before/after changes, use statistical significance testing (t-test), track metrics over time, and ensure benchmark diversity across difficulty levels and task types. Calculate effect sizes to determine if improvements are meaningful.
Q8: How do you evaluate agent safety and hallucination rates? A: Include adversarial test cases, measure factuality against ground truth, track citation accuracy, implement safety classifiers, and maintain a safety benchmark separate from performance benchmarks. Use LLM-as-judge for subjective quality assessments.
Common Pitfalls & Solutions
| Pitfall | Impact | Solution |
|---|---|---|
| Flaky tests | Unreliable results | Use deterministic test cases with temperature=0, average across runs |
| Metric overfitting | Misleading improvements | Use multiple complementary metrics, validate on holdout set |
| Evaluation bias | Unrepresentative results | Include diverse, representative test cases across demographics |
| Cost explosion | Budget overrun | Sample large test suites; use smaller models for testing |
| Stale test cases | Outdated benchmarks | Regular test case review and updates, version control |
| Ignoring edge cases | Missed failures | Include adversarial and boundary test cases |
| Single-run evaluation | Statistical noise | Average across multiple runs, compute confidence intervals |
| No baseline | Cannot measure improvement | Always maintain a baseline for comparison |
Summary with Key Takeaways
- Systematic evaluation enables data-driven agent improvement with reproducible metrics
- Execution tracing provides visibility into agent behavior and failure modes
- Multiple metrics (F1, latency, cost) capture different aspects of performance
- Comparative analysis quantifies improvement from changes with statistical significance
- Always evaluate safety and hallucination rates alongside performance metrics
- Maintain diverse, version-controlled test suites for consistent benchmarking