LLM Performance on 50 Advanced Cardiac Life Support Questions

Murat Genc's dataset from 2026 contains evaluation results for four large language models on a standardized medical assessment. The data includes performance metrics for 50 Advanced Cardiac Life Support questions across three categories, with each model tested three times for consistency. Results are provided in an XLSX file sized 12.1 KB.

Use Cases

• Compare LLM accuracy on medical knowledge questions based on the three defined accuracy metrics.
• Analyze model consistency in medical assessments based on the Fleiss' Kappa scores provided.
• Benchmark visual recognition capabilities of LLMs based on the visual content question category results.
• Evaluate the suitability of LLMs as supplementary tools in resuscitation education based on the comparative performance data.

Strengths

• Performance data for four distinct LLMs (ChatGPT-4o, Gemini 2.0, Claude 3.5, DeepSeek R1).
• Results for 50 questions categorized into three types (knowledge-based, visual content, case-based).
• Includes three accuracy metrics (overall, strict, ideal) and consistency scores (Fleiss' Kappa).
• Released under a permissive CC-BY-4.0 license.

Limitations

• Row count is unknown, which may limit suitability assessment.
• Column-level documentation is absent; field semantics must be inferred after download.
• The dataset is very small at 12.1 KB, indicating a limited scope of evaluation.

Provenance

Source

figshare

Collection Method

Observational study where 50 ACLS questions were posed to four LLMs on three separate occasions.

Freshness

Last updated 2026-04-29 17:32:52; freshness should be verified.