Diabetes Diagnosis Performance of Open-Source LLMs on Chinese Clinical Text

Shuting Yang's study evaluates open-source large language models on 11,329 adult diabetes patient records from a Chinese tertiary center (2010–2020). The analysis measures F1-scores for multi-class diabetes subtyping and binary classification of diabetic kidney disease and metabolic syndrome. It tests various model sizes, four prompting strategies, and performance in both English and Chinese.

Use Cases

• Analyze the impact of prompting strategies on F1-scores for diabetic kidney disease and metabolic syndrome classification.
• Compare the performance of English versus Chinese prompts on native Chinese clinical text for diabetes subtyping.
• Identify the optimal model size, such as 32B parameters, for achieving peak F1-scores in complex diabetes classification tasks.
• Investigate the trade-offs between Chain-of-Thought prompting on metabolic syndrome versus diabetic kidney disease diagnostic performance.

Strengths

• Based on a substantial cohort of 11,329 adult diabetes patient records.
• Evaluates performance across multiple specific clinical tasks: multi-class subtyping, diabetic kidney disease, and metabolic syndrome classification.
• Provides concrete performance metrics, including a peak F1-score of 0.951 for complex subtyping.

Limitations

• The dataset is a 24.7 KB document describing a study, not the underlying patient data, limiting direct analysis.
• Performance results are specific to a single Chinese tertiary center, which may limit generalizability.
• The study focuses on retrospective data from 2010–2020 and may not reflect current clinical practices or model capabilities.

Provenance

Source

Shuting Yang via figshare.

Collection Method

Retrospective analysis of unstructured clinical text from a large Chinese tertiary hospital.

Time Range

2010 to 2020.

Freshness

Last updated March 25, 2026.

Geography

A large Chinese tertiary medical center.