Medical Research AI Agent
Medical Research Agent Architecture
What is a Medical Research Agent?
Medical research agents automate literature review by searching PubMed, extracting key findings, summarizing papers, and synthesizing evidence across studies. They help researchers quickly understand the state of knowledge on a medical topic.
Why this matters: A systematic literature review can take weeks. A medical research agent can search 35M+ papers, extract relevant findings, and synthesize evidence in minutes.
Common Misconception
"AI can replace systematic review methodology."
AI agents accelerate the process but cannot replace systematic methodology. They help with search, extraction, and summarization â but study selection, quality assessment, and interpretation still require domain expertise.
Real-World Analogy
Think of it as a research librarian who can instantly search 35M papers, read abstracts in seconds, extract structured data, and synthesize findings â but who always reminds you to verify key findings against the original papers.
Project Overview
We will build a PubMed research agent that:
- Searches PubMed using the E-utilities API with MeSH terms
- Parses paper metadata and abstracts
- Extracts study design, sample size, and outcomes
- Summarizes key findings with evidence levels
- Manages citations in standard formats (AMA, APA, Vancouver)
- Generates literature review reports
Expected outcome: An agent that performs automated literature reviews from natural language queries.
Tools and Setup
| Tool | Version | Purpose |
|---|---|---|
| Python | 3.11+ | Core language |
| biopython | 1.80+ | PubMed E-utilities |
| requests | 2.31+ | HTTP client |
| openai | 1.0+ | LLM backbone |
| pydantic | 2.0+ | Data models |
Step 1: Environment Setup
python -m venv venv
source venv/bin/activate
pip install biopython requests openai pydantic
Step 2: PubMed Client
# pubmed/client.py
from Bio import Entrez
from typing import List, Dict, Optional
import time
import logging
logger = logging.getLogger(__name__)
Entrez.email = "research-agent@example.com"
class PubMedClient:
def __init__(self, api_key: Optional[str] = None):
if api_key:
Entrez.api_key = api_key
def search(self, query: str, max_results: int = 20) -> List[str]:
handle = Entrez.esearch(db="pubmed", term=query, retmax=max_results, sort="relevance", usehistory="y")
results = Entrez.read(handle)
handle.close()
return results.get("IdList", [])
def fetch_papers(self, pmids: List[str]) -> List[Dict]:
if not pmids:
return []
handle = Entrez.efetch(db="pubmed", id=",".join(pmids), rettype="xml", retmode="xml")
records = Entrez.read(handle)
handle.close()
return [self._parse_article(a) for a in records.get("PubmedArticle", [])]
def _parse_article(self, article: dict) -> Dict:
medline = article.get("MedlineCitation", {})
article_data = medline.get("Article", {})
abstract_parts = article_data.get("Abstract", {}).get("AbstractText", [])
abstract = " ".join(str(p) for p in abstract_parts)
authors = []
for author in article_data.get("AuthorList", []):
last, first = author.get("LastName", ""), author.get("ForeName", "")
if last:
authors.append(f"{last}, {first}")
return {
"pmid": str(medline.get("PMID", "")),
"title": article_data.get("ArticleTitle", "No title"),
"abstract": abstract,
"authors": authors,
"journal": article_data.get("Journal", {}).get("Title", ""),
"year": article_data.get("Journal", {}).get("JournalIssue", {}).get("PubDate", {}).get("Year", ""),
"keywords": [kw.strip() for kw in medline.get("KeywordList", [[]])[0]] if medline.get("KeywordList") else [],
}
def search_and_fetch(self, query: str, max_results: int = 10) -> List[Dict]:
pmids = self.search(query, max_results)
time.sleep(0.5)
return self.fetch_papers(pmids)
Step 3: Paper Parser and Summarizer
# processing/paper_parser.py
import json
from typing import Dict
from openai import OpenAI
class PaperParser:
def __init__(self, model: str = "gpt-4o"):
self.client = OpenAI()
self.model = model
def extract_structure(self, paper: Dict) -> Dict:
response = self.client.chat.completions.create(
model=self.model,
messages=[
{"role": "system", "content": """Extract structured info from this medical paper. Return JSON:
{"study_design": "RCT|cohort|case-control|cross-sectional|case-report|review|meta-analysis",
"sample_size": number or null, "population": "description",
"key_findings": ["list"], "limitations": ["list"],
"evidence_level": "1a|1b|2a|2b|3|4|5"}"""},
{"role": "user", "content": f"Title: {paper['title']}\nAbstract: {paper['abstract']}"},
],
temperature=0.0,
)
try:
return json.loads(response.choices[0].message.content)
except (json.JSONDecodeError, IndexError):
return {"study_design": "unknown", "evidence_level": "5"}
# processing/summarizer.py
from typing import Dict, List
from openai import OpenAI
class PaperSummarizer:
def __init__(self, model: str = "gpt-4o"):
self.client = OpenAI()
self.model = model
def summarize_paper(self, paper: Dict) -> str:
response = self.client.chat.completions.create(
model=self.model,
messages=[
{"role": "system", "content": "Summarize for a clinician. Include: objective, methods, key findings, implications. 150-200 words."},
{"role": "user", "content": f"Title: {paper['title']}\nAuthors: {', '.join(paper['authors'][:3])}\n{paper['abstract']}"},
],
temperature=0.2,
)
return response.choices[0].message.content
def synthesize_evidence(self, papers: List[Dict], question: str) -> str:
summaries = [f"- {p['title']} ({p['year']})\n {p.get('abstract', '')[:300]}" for p in papers[:10]]
response = self.client.chat.completions.create(
model=self.model,
messages=[
{"role": "system", "content": "Synthesize evidence. Provide: conclusion, consistency, gaps. Use GRADE-like language."},
{"role": "user", "content": f"Question: {question}\n\nStudies:\n" + "\n\n".join(summaries)},
],
temperature=0.3,
)
return response.choices[0].message.content
def format_citation(self, paper: Dict, style: str = "AMA") -> str:
authors = paper["authors"]
author_str = f"{authors[0]}, et al." if len(authors) > 3 else ", ".join(authors)
if style == "AMA":
return f"{author_str}. {paper['title']}. {paper['journal']}. {paper['year']}"
elif style == "APA":
return f"{author_str} ({paper['year']}). {paper['title']}. {paper['journal']}"
return f"{author_str}. {paper['title']}. {paper['journal']}. {paper['year']}"
Step 4: Complete Agent
# agent.py
from pubmed.client import PubMedClient
from processing.paper_parser import PaperParser
from processing.summarizer import PaperSummarizer
from typing import Dict
class MedicalResearchAgent:
def __init__(self, model: str = "gpt-4o", api_key: str = None):
self.pubmed = PubMedClient(api_key=api_key)
self.parser = PaperParser(model)
self.summarizer = PaperSummarizer(model)
def research(self, question: str, max_papers: int = 15) -> Dict:
papers = self.pubmed.search_and_fetch(question, max_papers)
analyzed = []
for paper in papers:
structure = self.parser.extract_structure(paper)
summary = self.summarizer.summarize_paper(paper)
analyzed.append({
**paper,
"structure": structure,
"summary": summary,
"citation_ama": self.summarizer.format_citation(paper, "AMA"),
"citation_apa": self.summarizer.format_citation(paper, "APA"),
})
synthesis = self.summarizer.synthesize_evidence(analyzed, question)
return {
"question": question,
"total_papers": len(analyzed),
"papers": analyzed,
"synthesis": synthesis,
"evidence_levels": [p["structure"].get("evidence_level", "5") for p in analyzed],
}
def get_paper_details(self, pmid: str) -> Dict:
papers = self.pubmed.fetch_papers([pmid])
if not papers:
return {"error": "Paper not found"}
paper = papers[0]
return {
**paper,
"structure": self.parser.extract_structure(paper),
"summary": self.summarizer.summarize_paper(paper),
}
Mathematical Foundation
Evidence Level Hierarchy (Oxford CEBM):
- 1a: Systematic review of RCTs (strongest evidence)
- 1b: Individual RCT
- 2a: Systematic review of cohort studies
- 2b: Individual cohort study
- 3: Case-control studies
- 4: Case series
- 5: Expert opinion (weakest evidence)
Heterogeneity Score (I-squared): I-squared > 75% indicates substantial heterogeneity limiting pooled estimates.
Performance Considerations
| Metric | Value | Notes |
|---|---|---|
| PubMed API Rate | 3 req/s (no key), 10 req/s (with key) | Rate limit is bottleneck |
| Cost per 10 Papers | $0.08 | ~4K tokens per paper |
| Extraction Accuracy | 90% | Study design + evidence level |
| Synthesis Quality | Good for abstracts | Limitations without full text |
Security Notes
- Store PubMed and OpenAI keys in environment variables
- Respect PubMed rate limits without API key (3 req/s)
- Agent output is for research only, not clinical decisions
- Note when analysis is based on abstracts only
- Never include patient data in prompts (HIPAA)
Interview Questions
1. How does the agent handle PubMed API rate limits?
PubMed allows 3 req/s without API key, 10 req/s with key. Implement time.sleep(0.5) between calls and exponential backoff on 429 errors. For large queries, use esearch history to paginate.
2. What is the difference between MeSH terms and free-text search?
MeSH terms are controlled vocabulary providing standardized search across synonyms. Free-text is faster but may miss synonyms. Use both: MeSH for recall, free-text for recent unindexed papers.
3. How does the agent assess evidence quality?
Beyond evidence level, evaluate: risk of bias, sample size adequacy, effect size, funding source, and follow-up duration. The GRADE framework provides systematic evidence certainty assessment.
4. How do you handle papers with only abstracts?
Clearly note when analysis is abstracts-only, avoid detailed methodology critique, focus on reported outcomes, and flag when full-text review is recommended.
5. How do you handle conflicting findings?
Identify contradictions by comparing effect directions and magnitudes. Note heterogeneity, explore explanations (different populations, interventions), assess which studies have higher evidence levels, and present conflicts transparently.
6. What are limitations of LLMs for evidence synthesis?
LLMs can hallucinate findings not in input, may misrepresent statistical results. Mitigate: only feed actual content, verify claims against source papers, use LLM as writing aid not analysis tool, have domain experts review outputs.
7. How would you extend for real-time monitoring?
Set up PubMed email alerts on schedule. Use vector database for previous summaries. Implement evidence update workflow that only re-summarizes papers with new findings.
8. What is the role of the GRADE framework?
GRADE assesses certainty of evidence across studies. It upgrades observational study evidence for large effects, dose-response, and plausible confounding. It downgrades for risk of bias, inconsistency, indirectness, imprecision, and publication bias.
Common Pitfalls and Solutions
| Pitfall | Solution |
|---|---|
| API rate limits | Implement delays; use API keys for higher limits |
| Abstract-only bias | Note limitations; avoid methodology critique |
| Language bias | Document exclusion of non-English papers |
| Publication bias | Search ClinicalTrials.gov for unpublished data |
| MeSH variation | Use both MeSH and free-text with boolean operators |
| LLM hallucination | Verify all claims against source abstracts |
| Outdated evidence | Filter by date; prioritize recent systematic reviews |
Summary with Key Takeaways
- PubMed E-utilities provides programmatic access to 35M+ medical papers
- Structured extraction enables quantitative evidence synthesis
- Evidence level classification helps assess study quality
- Automated summarization saves hours while maintaining accuracy
- Always note limitations including abstract-only analysis
- GRADE-based evidence certainty adds rigor to synthesis
- Citation management in multiple formats supports publication requirements
KnowledgeCheck
-
What evidence level does an individual RCT receive?
- a) 1a
- b) 1b
- c) 2a
- d) 3
-
What does I-squared of 80% indicate?
- a) High consistency
- b) Low heterogeneity
- c) Substantial heterogeneity limiting pooled estimates
- d) Perfect agreement
-
Why is an API key recommended for PubMed?
- a) Required for access
- b) Increases rate limits from 3 to 10 req/s
- c) Provides full-text access
- d) Enables MeSH search
-
What does "et al." indicate in AMA citation style?
- a) Foreign language paper
- b) More than 3 authors (only first listed)
- c) Systematic review
- d) No DOI
-
What is the primary advantage of MeSH over free-text?
- a) Faster search
- b) More papers
- c) Standardized terminology across synonyms
- d) Lower cost
-
Why should LLM outputs be verified against source papers?
- a) LLMs are always wrong
- b) LLMs may hallucinate findings not in source material
- c) LLMs can only read abstracts
- d) LLMs don't understand medical terminology
Answers: 1-b, 2-c, 3-b, 4-b, 5-c, 6-b