MedAgentBench: 300 FHIR-Based Clinical Agentic Tasks for EHR Workflows

MedAgentBench-Tasks consists of 300 FHIR-based agentic tasks across 10 clinical categories, released by Stanford-AIMI in 2025. It serves as the evaluation set for a clinical LLM benchmark designed to test models on real Electronic Health Record (EHR) workflows via a live FHIR REST API.

Use Cases

• Benchmarking LLM agent performance across 10 clinical task types such as patient data retrieval or clinical decision support
• Evaluating the ability of AI models to interact with FHIR REST APIs for medical data management
• Testing clinical workflow automation by executing the 300 provided task scenarios

Strengths

• 300 tasks categorized into 10 specific clinical workflow types
• Standardized FHIR-based task structure for interoperability testing
• Peer-reviewed methodology from Stanford-AIMI (arXiv:2501.07726)

Limitations

• Small scale with only 300 total records
• Dependency on a live FHIR REST API environment for full task execution
• Geographic bias toward US-centric clinical standards and FHIR implementations

Provenance

Source

Stanford-AIMI (arXiv:2501.07726)

Collection Method

Expert-designed clinical tasks

Time Range

2025

Freshness

Last updated February 2026; based on research published in early 2025.

Geography

US