Diabetes Diagnosis Performance of Open-Source LLMs on 11,329 Patient Records

A study evaluated open-source large language models (LLMs) for diagnosing diabetes subtypes and comorbidities from unstructured clinical text of 11,329 adult diabetes patients. The research assessed model performance using F1-scores for multi-class subtyping and binary classification of diabetic kidney disease and metabolic syndrome, testing various prompting strategies in English and Chinese.

Use Cases

• Analyze the correlation between LLM model size (e.g., 32B parameters) and F1-scores for diabetes subtyping and comorbidity classification.
• Compare the performance of English versus Chinese prompts on native Chinese clinical text for tasks like diabetic kidney disease diagnosis.
• Evaluate the impact of Chain-of-Thought prompting on F1-scores for metabolic syndrome classification versus rule-based diabetic kidney disease diagnosis.
• Assess the peak F1-score of 0.951 for complex diabetes subtyping against the lower scores for rule-based diagnostic tasks.

Strengths

• Contains performance metrics (F1-scores) for multiple diagnostic tasks from a study of 11,329 patient records.
• Evaluates a specific, impactful variable: the optimal LLM model size was found to be approximately 32B parameters.
• Compares results across two languages (English and Chinese) and four distinct prompting strategies.

Limitations

• The dataset is very small (65.3 KB), containing a study summary rather than the raw patient or model output data.
• Limited to a single tertiary center in China, which may restrict geographic and clinical practice generalizability.
• The core data (patient records, model outputs) are not directly accessible; only the study's analysis and conclusions are provided.

Provenance

Source

Retrospective analysis from a large Chinese tertiary center.

Collection Method

Evaluation of various open-source LLMs using prompting strategies on unstructured clinical text.

Time Range

Patient records from 2010 to 2020.

Freshness

null

Geography

Data from a tertiary center in China.