MMEB Eval: Massive Multimodal Embedding Benchmark with 36 Datasets

A benchmark for evaluating multimodal embedding models, covering 4 meta tasks and 36 datasets. The dataset was created by TIGER-Lab and published in the paper 'VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks'. It was last updated on Hugging Face on October 28, 2024.

Use Cases

• Benchmarking model performance on multimodal retrieval tasks based on the 4 meta tasks described.
• Evaluating embedding quality across diverse vision-language datasets based on the 36 included datasets.
• Training or fine-tuning multimodal models using the provided query-target evaluation examples.

Strengths

• Covers 4 meta tasks and 36 datasets for evaluation.
• Provides 1000 evaluation examples per dataset.
• Published in a peer-reviewed paper (VLM2Vec).

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.

Provenance

Source

TIGER-Lab

Collection Method

Compiled from a set of evaluation tasks, as described in the associated paper.

Freshness

Last updated 2024-10-28 16:42:34; freshness should be verified.