SynthDocBench: A Synthetic Benchmark for Vision-Language Models on Multi-Page PDFs

SynthDocBench is a fully synthetic benchmark for evaluating vision-language models on complex, multi-page PDF documents. It was created by ServiceNow-AI using an LLM pipeline to generate realistic reports with embedded D3.js charts and rich layouts. The dataset page was last updated on 2026-05-23.

Use Cases

• Benchmarking VLM performance on document question answering based on the description of deterministically grounded ground-truth answers.
• Evaluating model robustness to complex document layouts based on the mention of rich layouts and embedded charts.
• Controlled testing of multimodal reasoning on synthetic data to isolate specific capabilities, as the description notes it enables noise-free evaluation impossible with real-world corpora.

Strengths

• Provides deterministically grounded ground-truth answers, enabling controlled evaluation.
• Documents are generated with realistic multi-page reports, embedded D3.js charts, and rich layouts.
• The dataset is synthetic, which the description states enables noise-free evaluation impossible with real-world corpora.

Limitations

• Row count, file formats, and column-level documentation are unknown, which may limit suitability assessment.
• The data is fully synthetic, which may not capture all the complexities and noise of real-world documents.

Provenance

Source

ServiceNow-AI via Hugging Face.

Collection Method

Documents are generated end-to-end by an LLM pipeline.

Freshness

Last updated 2026-05-23 19:27:24.