CMRAG-Bench: A Multimodal Test Dataset for PDF Retrieval-Augmented Generation

CMRAG-Bench is a test dataset created for the CMRAG paper. All PDFs have been rendered and parsed by the Qwen2.5-VL-72B model. The dataset includes rendered page images, parsed HTML content, and JSON files containing sub-image bounding boxes.

Use Cases

• Benchmarking multimodal RAG systems based on the structured test data.
• Evaluating PDF parsing and visual document understanding models based on the rendered images and parsed content.
• Training or fine-tuning models for sub-image region detection based on the bounding box annotations.
• Researching document-to-image alignment techniques based on the paired PNG and HTML files.

Strengths

• All PDFs have been processed by a specific, high-capacity model (Qwen2.5-VL-72B).
• Data includes multiple synchronized representations: images, parsed text, and region metadata.

Limitations

• Description metadata is limited; actual data quality requires manual inspection after download.
• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.

Provenance

Source

huggingface

Collection Method

PDFs rendered and parsed by Qwen2.5-VL-72B.

Freshness

Last updated 2026-03-06 14:21:00; freshness should be verified.