MM-Vet: A Large-Scale Multi-Modality Model Evaluation Suite

A formatted evaluation suite for large multi-modality models (LMMs), created by lmms-lab and last updated on March 8, 2024. It is designed to accelerate LMM development by enabling one-click evaluations through the lmms-eval pipeline. The dataset is based on the MM-Vet benchmark described in the associated research paper.

Use Cases

• Benchmarking model performance on integrated multimodal capabilities based on the MM-Vet framework.
• Conducting one-click automated evaluations of large multi-modality models using the lmms-eval pipeline.
• Comparing different LMM architectures against a standardized set of tasks.

Strengths

• Specifically formatted for integration with the lmms-eval pipeline for streamlined evaluation.
• Based on the published MM-Vet benchmark, providing a structured evaluation framework.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count and dataset scale are unknown, which may limit suitability assessment.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

lmms-lab on Hugging Face, based on the MM-Vet research paper.

Collection Method

Formatted version of the MM-Vet benchmark for evaluation pipelines.

Time Range

The dataset page was updated in March 2024; the underlying benchmark's temporal coverage is not specified.

Freshness

Last updated 2024-03-08 05:02:09.

Geography

Not specified.