Olmocr Bench: OCR Evaluation Benchmark for PDF-to-Markdown Conversion

Olmocr Bench contains 1,403 PDF files and 7,010 unit test cases designed to evaluate the conversion of documents into markdown format. Created by allenai and updated in February 2026, this benchmark measures an OCR system's ability to preserve both textual content and structural properties.

Use Cases

• Evaluating OCR system accuracy using 7,010 unit test cases to verify structural integrity
• Benchmarking PDF-to-markdown conversion models against expert-defined properties
• Analyzing document layout preservation by testing for specific text presence in the output

Strengths

• Includes 7,010 unit test cases for granular performance tracking
• 1,403 source PDF files providing diverse document layouts
• Released under the ODC-BY license for open benchmarking

Limitations

• Small sample size of 1,403 files compared to general-purpose OCR training sets
• Geographic and linguistic bias toward English-language documents as indicated by metadata tags
• Evaluation is restricted to markdown output, which may not generalize to other structured formats

Provenance

Source

Allen Institute for AI (allenai), Arxiv paper 250218443

Freshness

Last updated February 19, 2026.

Geography

United States