OmniAI OCR Benchmark: 1,000+ Images for Multimodal LLM Evaluation

OmniAI OCR Benchmark provides between 1,000 and 10,000 image-text pairs for evaluating multimodal LLM performance, released by getomni-ai in February 2025. It measures the accuracy of text and JSON extraction from images, specifically targeting models like GPT-4o and Gemini 2.0.

Use Cases

• Benchmarking GPT-4o and Gemini 2.0 on text extraction accuracy
• Evaluating JSON schema adherence in multimodal LLM outputs
• Testing OCR accuracy on image-based documents

Strengths

• Contains 1,000 to 10,000 image-text pairs
• Evaluates both raw text and structured JSON extraction
• Released under the MIT license

Limitations

• Column names and specific schema details are not provided in the metadata
• Geographic scope is limited to the US region according to metadata tags

Provenance

Source

getomni-ai

Freshness

Last updated February 2025

Geography

United States