Medical Multimodal Evaluation Data: 6 Aggregated VQA Benchmarks

Aggregating 10,000 to 100,000 medical image-text pairs, this 2024 release from FreedomIntelligence serves as a standardized evaluation suite for multimodal LLMs. It incorporates six distinct benchmarks including VQA-RAD, SLAKE, and PathVQA to test models like HuatuoGPT-Vision.

Use Cases

• Evaluating zero-shot medical reasoning using the VQA-RAD and SLAKE subsets
• Testing pathological image interpretation via PathVQA image-text pairs
• Benchmarking multimodal medical knowledge using MMMU-Medical-Tracks

Strengths

• Aggregates 6 distinct medical benchmarks into a unified format
• Scale of 10,000 to 100,000 evaluation records
• Apache 2.0 licensed for open research

Limitations

• Designed for evaluation rather than large-scale model training
• Potential domain inconsistency between the six different source benchmarks

Provenance

Source

FreedomIntelligence (HuatuoGPT-Vision project)

Collection Method

Aggregation of existing open-source medical benchmarks including VQA-RAD, SLAKE, PathVQA, PMC-VQA, OmniMedVQA, and MMMU

Freshness

Last updated January 2025; reflects current medical VQA benchmarking standards.