Med-HallMark: Medical Multimodal Hallucination Benchmark

Med-HallMark is a benchmark dataset containing 750 image-question pairs for evaluating hallucinations in medical vision-language models. It includes three task types: conventional hallucination detection (499 pairs), counterfactual prompt-induced hallucination (111 pairs), and confidence weakening hallucination (140 pairs). The dataset was created by MM-Hallu and last updated on April 30, 2026.

Use Cases

• Benchmarking model hallucination rates based on conventional detection tasks
• Evaluating model robustness based on counterfactual prompt-induced hallucination tasks
• Assessing model confidence calibration based on confidence weakening hallucination tasks
• Training or fine-tuning models to reduce hallucinations in medical question-answering

Strengths

• Contains 750 specifically curated image-question pairs for hallucination evaluation
• Structured into three distinct task types with defined sample counts (499, 111, 140)
• Images are sourced from established medical datasets VQA-RAD and SLAKE

Limitations

• Description metadata is limited; actual data quality requires manual inspection after download
• Column-level documentation is absent; field semantics must be inferred after download
• An Image Report Generation (IRG) task with 359 records is noted as excluded due to licensing restrictions

Provenance

Source

Images sourced from VQA-RAD and SLAKE medical datasets.

Freshness

Last updated 2026-04-30 03:07:08; freshness should be verified