OlmOCR Bench 1.5 Preview: Synthetic Benchmark for Difficult OCR Scenarios

A preview version of the olmOCR benchmark, last updated March 2026 by AllenAI. It builds upon the original benchmark's 1,403 PDF files and 7,010 manually created unit test cases by adding new synthetic tests. These synthetic tests are designed to evaluate optical character recognition performance on challenging scenarios.

Use Cases

• Benchmarking OCR model robustness based on the described difficult synthetic scenarios.
• Evaluating model performance on a mix of real and synthetic PDF documents as mentioned in the description.
• Training OCR models to handle edge cases suggested by the synthetic test categories.
• Comparing OCR tools using the manually created unit test cases included from the original benchmark.

Strengths

• Includes 1,403 original PDF files providing a base of real document data.
• Contains 7,010 manually created unit test cases for precise evaluation.
• Adds new synthetic tests designed to target difficult OCR scenarios.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

AllenAI

Collection Method

Combines manually created unit tests from an original benchmark with new synthetically generated tests.

Time Range

null

Freshness

Last updated 2026-03 23 20:09:33; freshness should be verified.

Geography

null