Multimodal Embedding Benchmark for Video and Document Tasks

MMEB-V2 is a benchmark dataset for evaluating multimodal embedding models, created by VLM2Vec and updated in September 2025. It expands upon a previous version to include five new tasks: Video Retrieval, Moment Retrieval, Video Classification, Video Question Answering, and Visual Document Retrieval.

Use Cases

• Benchmark multimodal models on Video Retrieval and Video Question Answering tasks using video-based data.
• Evaluate embedding performance on Moment Retrieval, which requires temporal localization within videos.
• Test model capabilities on Visual Document Retrieval, a task focused on structured visual data.
• Assess model generalization across static, temporal, and structured visual data settings as defined by the benchmark suite.

Strengths

• Benchmark includes five distinct evaluation tasks for multimodal models.
• Dataset is an updated version (V2) building upon a prior benchmark, indicating iterative development.
• Last updated in September 2025, suggesting recent maintenance.

Limitations

• Specific dataset size, row count, and column structure are unknown.
• The raw description is truncated, limiting detailed understanding of data composition and sample availability.
• Lack of provided sample data or file formats makes preliminary inspection impossible.

Provenance

Source

Hugging Face dataset authored by VLM2Vec.

Collection Method

null

Time Range

null

Freshness

Last updated on 2025-09-24.

Geography

null