MediQ: Clinical Reasoning Benchmark with 1,000-10,000 Interactive Scenarios

MediQ contains between 1,000 and 10,000 text-based records designed to benchmark interactive clinical reasoning in Large Language Models. Released in 2024 by researcher stellalisy, the dataset evaluates how models identify missing information and ask clarifying questions in medical contexts. It is formatted for use with common data libraries like Pandas and Polars.

Use Cases

• Evaluating LLM 'question-asking' strategies within clinical scenarios
• Benchmarking interactive reasoning performance using the provided JSON format
• Training models to identify missing diagnostic information in medical text

Strengths

• 1,000 to 10,000 record scale
• CC BY 4.0 open license
• Linked to Arxiv 2406.00922 research

Limitations

• Domain-specific focus on clinical reasoning limits generalizability to non-medical tasks
• Potential for synthetic bias if scenarios were generated or augmented by other LLMs

Provenance

Source

Arxiv 2406.00922: MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning

Collection Method

Annotated or synthetic generation for benchmarking purposes

Freshness

Last updated July 2025; based on research published in June 2024.

Geography

United States