Thyroid Ultrasound Interpretation Performance of Four Large Language Models

538.3 KB of research data evaluates four CoT-enabled LLMs (ChatGPT-O3, Grok-3, DeepSeek-R1, Gemini-2.5 Pro) on thyroid nodule interpretation. The study assesses diagnostic accuracy, reproducibility, and reasoning authenticity for both qualitative ACR-TIRADS features and quantitatively encoded ultrasound descriptors. Results show Grok-3 achieved 96% accuracy in qualitative tasks, while cross-modal conflicts ranged from 27 to 36% across models.

Use Cases

• Compare ACR-TIRADS categorization accuracy metrics (e.g., 96% for Grok-3) across the four evaluated LLMs.
• Analyze cross-threshold discrepancy rates (3-17%) and cross-modal conflict rates (27-36%) to assess diagnostic consistency.
• Evaluate reasoning authenticity and conciseness scores (e.g., 144 mean words for ChatGPT-O3) assigned by radiologists.
• Benchmark quantitative task accuracy (e.g., 78% for Gemini-2.5 Pro) and reproducibility (e.g., 86% for DeepSeek-R1) for encoded ultrasound features.

Strengths

• Provides performance metrics for four distinct LLMs (ChatGPT-O3, Grok-3, DeepSeek-R1, Gemini-2.5 Pro) on the same tasks.
• Includes specific accuracy percentages (e.g., 96%, 79%) and reproducibility rates (e.g., 93%, 18%) for both qualitative and quantitative analysis.
• Assesses two distinct inconsistency types: cross-threshold discrepancies (3-17%) and cross-modal conflicts (27-36%).

Limitations

• The dataset is a 538.3 KB DOCX file containing summarized results, not the underlying raw data or structured prompts used in the study.
• Sample size and specific row/column counts for the original nodule evaluations are not provided in the available description.
• The data is retrospective and based on a specific methodological setup, limiting direct generalization to other LLMs or clinical contexts.

Provenance

Source

figshare, authored by Yu-Tong Zhang.

Collection Method

Retrospective study analyzing thyroid nodules with radiologist-labeled qualitative features and quantitatively encoded descriptors, converted to structured prompts for LLM evaluation.

Freshness

Last updated on 2026-03-23.