MME-RealWorld: A Multimodal Benchmark for Large Language Model Evaluation

MME-RealWorld is a multimodal benchmark dataset for evaluating large language models, launched on August 20, 2024. The dataset includes a lite version with 50 samples per task for inference acceleration, as noted in the November 14, 2024 update. It is authored by yifanzhang114 and is supported by evaluation frameworks like VLMEvalKit and Lmms-eval.

Use Cases

• Benchmarking model performance on visual question answering tasks based on the dataset's multimodal nature.
• Accelerating inference testing using the lite version with 50 samples per task.
• Comparing model accuracy on a leaderboard, as the description notes top-performing models have overall accuracy below 55%.
• Conducting one-click evaluations by integrating with supported frameworks like VLMEvalKit.

Strengths

• Includes a lite version with 50 samples per task for faster inference testing.
• Integrated with established evaluation frameworks (VLMEvalKit, Lmms-eval) for streamlined use.
• Maintains a public leaderboard for tracking model performance, with results updated as of October 27, 2024.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count and total dataset size are unknown, which may limit suitability assessment.
• The overall model accuracy on the benchmark, as of October 2024, remains below 55%, indicating challenging tasks.

Provenance

Source

huggingface

Collection Method

Likely curated for benchmark evaluation, but specific collection method is not detailed.

Time Range

Dataset activity and updates span from August to November 2024.

Freshness

Last updated 2024-11-14 02:44:38.

Geography

Region tags suggest a US focus, but specific spatial coverage is unknown.