MedMisBench: A Medical QA Benchmark for LLMs with Misleading Context

MedMisBench evaluates whether large language models preserve correct medical judgment when misleading context is introduced. The benchmark is built from five medical question-answering sources spanning standard medical reasoning, expert reasoning, patient-journey scenarios, and agentic biomedical capability. It was created by HongjianZhou and last updated on June 15, 2026.

Use Cases

• Benchmarking LLM robustness based on multiple-choice medical questions with misleading context.
• Studying failure modes in medical reasoning based on the five source types mentioned in the description.
• Developing mitigation strategies for AI in healthcare based on adversarial medical QA scenarios.

Strengths

• Benchmark is built from five distinct medical question-answering sources.
• Focuses on a specific evaluation task: preserving judgment against misleading medical context.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Last updated 2026-06-15 04:40:37; freshness should be verified.

Provenance

Source

HongjianZhou on Hugging Face.

Collection Method

Built from five medical question-answering sources.

Freshness

Last updated 2026-06-15 04:40:37.