IDSA/ATS Criteria Agreement Analysis by Human Experts and AI Models

Agreement analysis compares scoring decisions by two independent internists against two AI models (NLP-V1 and V2). The dataset is 5.5 KB in size, authored by Alejandro Hernández-Arango, and was last updated on April 30, 2026. All False Positives are reported as zero across all criteria due to a conservative regex design.

Use Cases

• Benchmarking NLP model performance against human expert consensus based on the described agreement analysis.
• Studying the impact of conservative regex design on false positive rates in clinical text classification.
• Analyzing inter-rater reliability for IDSA/ATS criteria scoring between human experts.

Strengths

• Reported zero false positives across all criteria due to a specific conservative regex design.
• Includes agreement metrics from two independent human experts and two distinct AI model versions.
• Dataset is openly licensed under CC-BY-4.0.

Limitations

• The dataset is very small at 5.5 KB, indicating limited scope.
• Row count and column-level documentation are unknown, which may limit suitability assessment.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

figshare

Collection Method

Analysis of agreement between human experts and AI models on scoring IDSA/ATS criteria.

Freshness

Last updated 2026-04-30 17:50:06; freshness should be verified.