Blinded Two-Phase Evaluation of LLMs in Complex Cardiac Surgery

A research paper presents a two-phase evaluation framework for large language models in cardiac surgery. The study includes 15 high-fidelity clinical scenarios developed by senior surgeons and evaluates five LLMs using a 10-dimensional weighted framework. The dataset, a 369.8 KB PDF, was authored by Marc Leon and last updated on 2026-05-29.

Use Cases

• Benchmarking LLM performance on complex, open-ended clinical reasoning tasks based on the 15 cardiac surgery scenarios.
• Studying human-AI collaboration patterns based on the two-phase blinded evaluation and rating revision process.
• Analyzing model weaknesses in specific clinical dimensions like patient safety and hallucination avoidance based on the 10-dimensional evaluation framework.

Strengths

• Evaluation is based on 15 high-fidelity cardiac surgery scenarios developed independently by senior surgeons.
• Includes performance scores for five specific LLMs (O1, O3-mini-high, DeepSeek-R1, GPT-4, Llama3-OpenBioLLM-70B) across a 10-dimensional framework.
• Documents a two-phase blinded evaluation process with a second round where 7.57% of ratings were revised from affirmative to negative.

Limitations

• The primary data file is a 369.8 KB PDF; underlying evaluation data or model outputs are not provided as structured data.
• Row count and column-level documentation for any underlying data are unknown.
• The dataset's scope is limited to the specific evaluation described in the paper.

Provenance

Source

figshare

Collection Method

A panel of senior cardiac surgeons developed scenarios; a separate group conducted blinded evaluations of LLM responses.

Freshness

Last updated 2026-05-29 06:11:59; freshness should be verified.