CC-OCR Benchmark for Large Multimodal Model Literacy Evaluation

CC-OCR is a benchmark dataset for evaluating large multimodal models in optical character recognition and literacy tasks. The dataset is hosted in a TSV format for use with the VLMEvalKit evaluation framework. It was created by author wulipc and last updated in December 2024.

Use Cases

• Benchmark large multimodal models on OCR and literacy tasks using the provided evaluation code.
• Evaluate model performance on text extraction from images and documents as defined by the CC-OCR benchmark.
• Use the TSV-formatted data for structured testing within the VLMEvalKit framework.

Strengths

• Designed as a comprehensive and challenging benchmark specifically for evaluating OCR in large multimodal models.
• The dataset is integrated with the VLMEvalKit, providing a standardized evaluation framework.
• Maintained with a recent update in December 2024.

Limitations

• Specific dataset size, row count, column structure, and sample data are not provided in the input.
• The dataset's primary purpose is for benchmarking, which may limit its utility for general model training.
• Geographic and temporal coverage of the source images or documents is unknown.

Provenance

Source

Hugging Face dataset repository under author wulipc.

Collection Method

Created as part of the research for the paper 'CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy'.

Freshness

Last updated on 2024-12-27.