BigDocs-Bench: Benchmark for Multimodal Document and Code Tasks

BigDocs-Bench is a benchmark suite introduced by ServiceNow for evaluating multimodal models on tasks that transform visual inputs into structured outputs. The dataset is associated with the paper 'BigDocs: An Open and Permissively-Licensed Dataset for Training Multimodal Models on Document and Code Tasks'. The benchmark data was initially released on 2024-12-10 and last updated on the platform on 2025-03-19.

Use Cases

• Benchmarking model performance on GUI-to-user-intent tasks based on the description of fine-grained visual input transformation.
• Evaluating multimodal models on document understanding tasks based on the dataset's stated purpose.
• Training and testing models for generating structured outputs from visual inputs based on the benchmark's design focus.

Strengths

• Dataset is associated with a peer-reviewed arXiv paper, providing academic context.
• Benchmark is designed for specific, fine-grained downstream tasks like GUI2UserIntent.
• Last platform update was on 2025-03-19, indicating recent maintenance.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count, file formats, and total size are unknown, which may limit suitability assessment.
• The description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

ServiceNow

Collection Method

Likely created as a benchmark for the associated research paper.

Time Range

Initial release 2024-12-10.

Freshness

Last updated 2025-03 19 16:37:22; freshness should be verified.

Geography

Regionus tag suggests a US focus, but specific coverage is unknown.