Key Takeaways
- DeepEvidence is a biomedical deep research agent that explores knowledge graphs instead of web text
- Architecture: one orchestrator + two search modes — BFRS (breadth-first) and DFRS (depth-first)
- Beat every baseline on 4 open benchmarks: HLE-Medicine 40.0%, LabBench-LitQA2 80.0%, SuperGPQA-Medicine-Hard 47.1%, TrialPanorama-EvidenceQA 96.0%
- Beat Biomni and ToolUniverse on all 7 newly curated discovery-lifecycle tasks
- Published in Nature Machine Intelligence (2026); code ships inside the BioDSA framework
- Eval dataset is open: 159 rows, Apache 2.0, on Hugging Face
What Is DeepEvidence?
DeepEvidence is an AI agent system for deep research over heterogeneous biomedical knowledge graphs.
Core attributes:
- Paper: Empowering biomedical evidence exploration and synthesis with deep knowledge graph research, Nature Machine Intelligence, 2026
- Preprint: arXiv:2601.11560, submitted 23 December 2025
- Authors: Zifeng Wang, Zheng Chen, Ziwei Yang, Xuan Wang, Qiao Jin, Yifan Peng, Zhiyong Lu, Jimeng Sun
- Code: ships as one of 8 agents inside BioDSA
- Dataset: zifeng-ai/DeepEvidence — 159 rows, 362 kB, Apache 2.0, evaluation-only
- Knowledge bases: 16 in the tool library
- Backbone in all reported runs: GPT-5
The problem it targets:
- Most deep research agents read internet-scale text corpora only
- Biomedical knowledge lives in structured graphs with incompatible schemas
- Cross-resource entity alignment is weak, so multi-hop evidence chains break
- Generic agents with large tool suites suffer tool confusion — picking irrelevant tools and following wrong reasoning paths
How the Architecture Works
Four components:
| Component | Function |
|---|---|
| Orchestrator agent | Plans the research, picks the sequence of BFRS/DFRS steps, sets research targets |
| BFRS agent | Breadth-First ReSearch — broad multi-graph queries over first-hop neighborhoods |
| DFRS agent | Depth-First ReSearch — multi-hop tracing from entities BFRS surfaced |
| Evidence graph | Internal memory, updated incrementally, records entities/relations/supporting evidence |
Why the naming departs from BFS/DFS: the agents choose traversal steps autonomously rather than following fixed graph-theoretic rules. They also bridge across heterogeneous graphs by recognizing shared entities — called bridge entities in the paper.
Execution sandbox:
- Agents write and execute code to query KG APIs at scale
- Retrieved results persist as structured files (e.g. spreadsheets)
- Downstream extraction and analysis run inside the same controlled environment
- One drug query can fan out to KEGG, ChEMBL, OpenFDA and Open Targets in parallel
The 16 knowledge bases in the tool library: BioThings, ChEMBL, ClinicalTrials.gov, Gene Ontology, Human Phenotype Ontology, KEGG, NCBI Datasets API, OpenFDA, Open Genes, ProteinAtlas, PubChem, PubTator, PubMed, Reactome, UMLS, UniProt. Open Targets is additionally queried by the unified gene and drug search tools.
Benchmark Results: 4 Open Benchmarks
All agent systems ran on the same backbone (GPT-5). LLM baselines were equipped with literature search.
| Benchmark | Set size | DeepEvidence | Biomni | ToolUniverse | Sonnet-4.5 | GPT-5 |
|---|---|---|---|---|---|---|
| HLE-Medicine | 30 questions | 40.0% | 20.0% | 10.0% | 3.3% | 3.3% |
| LabBench-LitQA2 | 25 questions | 80.0% | 32.0% | n/r | 48.0% | n/r |
| SuperGPQA-Medicine-Hard | 172 questions | 47.1% | 40.7% | n/r | 43.6% | n/r |
| TrialPanorama-EvidenceQA | 50 questions | 96.0% | 84.0% | n/r | 88.0% | n/r |
n/r = value not reported in the paper's running text.
How the evaluation sets were built:
- HLE-Medicine: HLE questions labeled "Medicine", image-input questions removed → 30 questions
- LabBench-LitQA2: 25 questions randomly sampled from the LitQA2 subset
- SuperGPQA-Medicine-Hard: difficulty "Hard" ∩ field "Clinical Medicine" → 172 questions
- TrialPanorama-EvidenceQA: 50 most recent EvidenceSummary questions, with the supplied abstracts stripped from the input — forcing agents to find the literature themselves
That last modification is the most informative one. Removing the abstracts converts a summarization task into a retrieval-plus-reasoning task, and the 96.0% vs 88.0% gap over Sonnet-4.5 is measured under that harder setup.
Benchmark Results: 7 Discovery-Lifecycle Tasks
The authors curated new tasks spanning drug discovery, preclinical research, clinical trial development, and evidence-based medicine.
| Task | Metric | DeepEvidence | Biomni | ToolUniverse | LLM |
|---|---|---|---|---|---|
| Target identification | Accuracy | 68% | 56% | 40% | 42% |
| Mechanism-of-action & pathway reasoning | Accuracy | 72% | 44% | 44% | 52% |
| In vivo metabolic flux response | Accuracy | 80% | 68% | 52% | 60% |
| Sample size estimation | Accuracy | 68% | 20% | 32% | 24% |
| Drug regimen design | Accuracy | 52% | 36% | 28% | 20% |
| Surrogate endpoint discovery | F1 | 73.3% | 60.9% | 61.9% | 58.7% |
| Evidence gap discovery | Gap detection rate | 90.0% | 50.0% | 15.0% | 10.0% |
| Evidence gap discovery | Recall | 44.14% | 30.17% | 5.06% | 2.67% |
Surrogate endpoint discovery, full breakdown:
- DeepEvidence: 73.2% precision / 75.0% recall / 73.3% F1
- Biomni: 60.8% / 64.3% / 60.9%
- ToolUniverse: 57.7% / 69.0% / 61.9%
- LLM: 58.4% / 61.9% / 58.7%
How the hardest tasks were constructed:
- Target identification: 85 human KEGG disease pathways parsed from KGML into directed signaling graphs; ground truth comes from a topological logic engine scoring path polarity, rather than manual annotation; 10 candidate genes per question
- Sample size estimation: the 25 most recent trials from TrialPanorama's sample-size instances
- Evidence gap discovery: the agent receives a systematic review's question, objectives and eligibility criteria, then must surface trials the original review missed; scored on the top 30 proposals
The two biggest margins are worth naming. Sample size estimation: 68% against Biomni's 20%. Evidence gap detection: 90.0% against ToolUniverse's 15.0%. Both are tasks where the answer is only reachable by actually reading a lot of prior trials.
Why It Beats Biomni and ToolUniverse
The paper's own explanation is about search depth, not tool count.
| Dimension | DeepEvidence | Biomni / ToolUniverse |
|---|---|---|
| KG API access | 16 knowledge bases | Many KG APIs available |
| Search behavior | Structured BFRS→DFRS traversal | Few searches before answering |
| Memory | Continually updated evidence graph | No persistent evidence structure |
| Tool selection | Domain-specific guidance | Large heterogeneous tool suite → tool confusion |
| Failure mode observed | — | Shallow retrieval, irrelevant tool picks, noisy tool outputs |
| Traceability | Evidence graph inspectable by humans | Reasoning trace only |
Both Biomni and ToolUniverse already reach many of the same APIs. The measured difference comes from how many hops the agent takes before it answers, and whether it keeps a structured record of what it found.
Which Agent Should You Use?
Decision tree:
- Evidence synthesis, systematic review updating, trial design → DeepEvidence, because gap detection (90.0%) and EvidenceQA (96.0%) are its strongest measured axes
- General biomedical task coverage across many modalities → Biomni, which targets breadth over depth
- Tool-calling layer for your own reasoning model → ToolUniverse, 600+ scientific tools and MCP servers
- Building your own biomedical agent → BioDSA, which ships DeepEvidence plus 7 other published agents and skill packs for Claude Code and OpenClaw
- Benchmarking a hypothesis-validation agent → BioDSA-1K, 1,029 tasks over patient-level cBioPortal data
By role:
Clinical evidence teams:
- Recommended: DeepEvidence
- Key reason: 90.0% gap detection rate on newly relevant trials, versus 10.0% for a PubMed-search LLM
- Caveat: recall is 44.14%, so it surfaces candidates for human screening rather than a complete set
Drug discovery teams:
- Recommended: DeepEvidence for target identification (68%) and MoA reasoning (72%)
- Key reason: it grounds gene identities across databases before reasoning
- Caveat: drug regimen design sits at 52%, the lowest of the seven tasks
Stated Limitations
The authors are direct about what the system cannot yet do:
- Coverage is incomplete — the benchmarks reflect a subset of biomedical research tasks
- Works on well-curated graphs — KEGG and Gene Ontology offer stable schemas and high-quality APIs; proprietary or internally hosted graphs with missing values and inconsistent structure would require the agent to construct new tools
- Two traversal strategies only — BFRS and DFRS improve over generic agents, and the authors state they are unlikely to be optimal
- Below expert level on many tasks — 52% on drug regimen design and 44.14% recall on evidence gaps are the clearest examples
- Text-only — imaging, structural biology and proteomic modalities are future work
FAQ
Q1: What exactly is DeepEvidence?
DeepEvidence is an AI agent framework for deep research over biomedical knowledge graphs. It uses an orchestrator agent that delegates to breadth-first (BFRS) and depth-first (DFRS) research sub-agents, and maintains an internal evidence graph recording entities, relations and supporting evidence throughout the search.
Q2: Where was DeepEvidence published?
In Nature Machine Intelligence (2026), under the title "Empowering biomedical evidence exploration and synthesis with deep knowledge graph research". The preprint is arXiv:2601.11560, submitted 23 December 2025. Affiliations span Keiji AI, Osaka University, Kyoto University, the NIH National Library of Medicine, Weill Cornell Medicine and UIUC.
Q3: How does DeepEvidence compare to Biomni?
DeepEvidence outperformed Biomni on all four open benchmarks and all seven curated tasks, with the same GPT-5 backbone. Largest gaps: LabBench-LitQA2 80.0% vs 32.0%, sample size estimation 68% vs 20%, evidence gap detection 90.0% vs 50.0%. The paper attributes the difference to structured deep traversal plus evidence-graph memory.
Q4: Is DeepEvidence open source?
The agent implementation ships inside the BioDSA framework at github.com/RyanWangZf/BioDSA under MIT. The evaluation dataset is on Hugging Face as zifeng-ai/DeepEvidence under Apache 2.0 — 159 rows across seven task families, evaluation-only.
Q5: Which knowledge bases does DeepEvidence query?
Sixteen: BioThings, ChEMBL, ClinicalTrials.gov, Gene Ontology, Human Phenotype Ontology, KEGG, NCBI Datasets API, OpenFDA, Open Genes, ProteinAtlas, PubChem, PubTator, PubMed, Reactome, UMLS and UniProt. Open Targets is additionally queried by the unified gene and drug search tools.
Q6: Can I run DeepEvidence on my own data?
The current system is designed for well-curated public graphs with stable schemas and high-quality APIs. The authors identify proprietary or internally hosted graphs as future work requiring the agent to construct new access tools.
Q7: What is the difference between DeepEvidence and BioDSA-1K?
DeepEvidence is an agent; BioDSA-1K is a benchmark. Both come from the same group. DeepEvidence performs knowledge-graph deep research; BioDSA-1K scores agents on 1,029 hypothesis-validation tasks grounded in patient-level cBioPortal data. Both are distributed through the BioDSA framework.
Q8: Does DeepEvidence replace human systematic reviewers?
Its recall on evidence gap discovery is 44.14%, meaning it surfaces under half of the true positives within its top-30 proposals. That positions it as a screening accelerator with human verification, and the evidence graph exists precisely so intermediate relations can be inspected.
Summary
DeepEvidence builds its deep research loop directly on biomedical knowledge graphs, where generic deep research systems rely on internet-scale text.
Core findings:
- Depth beats tool count: with an identical GPT-5 backbone, structured BFRS→DFRS traversal plus evidence-graph memory outperformed agents that had comparable API access
- Largest measured margins: LabBench-LitQA2 80.0% (2.5× Biomni), evidence gap detection 90.0% (6× ToolUniverse), sample size estimation 68% (3.4× Biomni)
- Reproducible: MIT-licensed code in BioDSA, Apache 2.0 evaluation dataset on Hugging Face
- Honest ceiling: 52% on drug regimen design and 44.14% recall on evidence gaps place it below expert level on the hardest tasks
Both DeepEvidence's home framework and its sibling benchmark are now in the directory — BioDSA under Bioinformatics & Omics, BioDSA-1K under Benchmarks & Evaluation.
Sources
- Empowering biomedical evidence exploration and synthesis with deep knowledge graph research — Nature Machine Intelligence, 2026
- arXiv:2601.11560 — DeepEvidence preprint
- zifeng-ai/DeepEvidence — evaluation dataset
- RyanWangZf/BioDSA — framework and agent implementation
- arXiv:2505.16100 — BioDSA-1K benchmark