MedPI: 7,097 Medical AI Conversations for Multi-Turn Evaluation

TheLumos created MedPI-dataset, a collection of 7,097 simulated medical conversations between AI models and synthetic patients across various specialties. Each conversation is assessed across up to 105 dimensions, including 46 global core competencies and 59 encounter-specific competencies, as outlined in the MedPI Eval framework. The dataset was last updated on December 17, 2025.

Use Cases

• Benchmarking AI model performance in simulated patient consultations based on the multi-turn conversation structure.
• Evaluating clinical competency across 105 dimensions as defined by the MedPI framework.
• Training or fine-tuning conversational AI models for medical specialties mentioned in the description.
• Analyzing the quality of AI-generated clinical advice in patient-facing scenarios.

Strengths

• Contains 7,097 distinct medical conversations, providing a substantial corpus for evaluation.
• Assesses conversations across up to 105 detailed competency dimensions.
• Covers various medical specialties, likely offering breadth across clinical domains.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is known, but specific file formats, sample data, and license information are unavailable.
• Data consists of synthetic patient conversations, which may not fully capture real-world clinical interactions.

Provenance

Source

TheLumos on Hugging Face.

Collection Method

Simulated conversations between AI models and synthetic patients.

Freshness

Last updated 2025-12-17 19:15:38; freshness should be verified.