PI-RADS Score Extraction from Prostate MRI Reports: LLM vs. Human Reader Comparison

A comparative study published in 2026 evaluates GPT-4o and Gemini 2.5 Pro for extracting PI-RADS v2.1 scores from free-text prostate MRI reports. The dataset likely contains the processed reports and assigned scores used to compare LLM performance against three human radiologists of varying experience. Author Jing Wen released the study document under a CC-BY-4.0 license.

Use Cases

• Benchmark LLM performance for clinical information extraction based on PI-RADS score assignment
• Analyze inter-rater agreement in medical text interpretation based on Gwet's AC1 coefficients
• Compare diagnostic performance metrics (AUC, sensitivity, specificity) between AI and human readers

Strengths

• Includes performance metrics with confidence intervals (e.g., AUCs ranging from 0.81 to 0.89)
• Compares three distinct human reader experience levels (resident, fellow, expert)
• Evaluates two specific LLM models (GPT-4o and Gemini 2.5 Pro)

Limitations

• Row count is unknown, which may limit suitability assessment
• Column-level documentation is absent; field semantics must be inferred after download
• The dataset is very small (17.9 KB), indicating limited scope

Provenance

Source

figshare

Collection Method

Likely contains data from a comparative study where radiologists and LLMs reviewed prostate MRI reports.

Freshness

Last updated 2026-04-10 09:36:02