Diabetes Diagnosis Performance of Open-Source LLMs on 11,329 Patient Records

Shuting Yang's 2026 study evaluates open-source large language models for diagnosing diabetes subtypes and comorbidities from unstructured clinical text. The analysis uses 11,329 adult diabetes patient records from a Chinese tertiary center spanning 2010-2020. It measures model performance via F1-scores for multi-class subtyping and binary classification of diabetic kidney disease and metabolic syndrome.

Use Cases

• Analyze the relationship between model size (e.g., 32B parameters) and F1-scores for diabetes subtyping and comorbidity classification.
• Compare the performance of English versus Chinese prompting strategies on native Chinese clinical text for diagnostic tasks.
• Evaluate the impact of Chain-of-Thought prompting on F1-scores for metabolic syndrome versus diabetic kidney disease classification.
• Benchmark open-source LLMs on rule-based diagnostic tasks (DKD, MetS) versus complex holistic pattern recognition for diabetes subtyping.

Strengths

• Analysis is based on a substantial cohort of 11,329 adult diabetes patient records.
• Provides specific performance metrics (F1-scores) for multiple diagnostic tasks, including a peak F1 of 0.951 for subtyping.
• Evaluates models across key variables: model size, prompting strategy (four types), and language (English/Chinese).
• Covers a decade of clinical data from 2010 to 2020.

Limitations

• The dataset is a 1.9 MB DOCX file containing a study report, not the underlying structured patient data or model outputs.
• Clinical data originates from a single Chinese tertiary center, limiting geographic and institutional generalizability.
• Performance results are specific to the tested open-source LLMs and may not reflect capabilities of other models or updated versions.

Provenance

Source

Shuting Yang via figshare.

Collection Method

Retrospective analysis of unstructured clinical text from electronic health records.

Time Range

2010 to 2020.

Freshness

Last updated March 25, 2026.

Geography

Patients from a large Chinese tertiary medical center.