Cardiac Surgery LLM Benchmark: Two-Phase Evaluation and Human-AI Collaboration

Marc Leon's dataset contains a two-phase evaluation of five large language models (O1, O3-mini-high, DeepSeek-R1, GPT-4, and Llama3-OpenBioLLM-70B) on 15 high-fidelity cardiac surgery scenarios. The data includes expert-curated reference answers and a 10-dimensional weighted evaluation framework, capturing model performance and shifts in surgeon ratings after seeing reference answers. The dataset is designed to assess LLM capabilities and human-AI collaboration in complex surgical decision-making.

Use Cases

• Benchmarking LLM performance on clinical reasoning tasks based on the 15 scenario-specific scores.
• Analyzing human-AI collaboration patterns based on the two-phase rating shifts (e.g., 7.57% revised from affirmative to negative).
• Comparing model strengths across evaluation dimensions like patient safety (0.507) and scenario comprehension (0.920).
• Studying evaluator judgment calibration after exposure to expert reference answers.

Strengths

• Contains expert-developed, high-fidelity clinical scenarios (15 scenarios) with reference answers.
• Uses a detailed, multi-dimensional evaluation framework (10 dimensions) with weighted scores.
• Captures a unique two-phase evaluation process, measuring rater judgment shifts.
• Released under a permissive CC-BY-4.0 license, facilitating reuse.

Limitations

• Column names and exact data structure are not specified in the provided metadata.
• The dataset size is reported inconsistently across entries (e.g., 681348, 115375, 10494), creating uncertainty about the total volume.
• The number of rows is not provided, limiting understanding of the dataset's granularity.

Provenance

Source

Marc Leon

Collection Method

A panel of senior cardiac surgeons developed scenarios and a separate group conducted a blinded two-phase evaluation of LLM outputs.

Freshness

Last updated 2026-05-29.