VS-Bench: Vision-Language Model Benchmark for Multi-Agent Environments

VS-Bench is a multimodal benchmark for evaluating Vision-Language Models in multi-agent environments. The benchmark evaluates fourteen state-of-the-art models across eight vision-grounded environments using two complementary dimensions. It was created by author zelaix and last updated on June 4, 2025.

Use Cases

• Benchmarking model performance on next-action prediction accuracy based on offline evaluation of strategic reasoning.
• Evaluating decision-making capabilities based on normalized episode return from online evaluation.
• Comparing Vision-Language Models across eight distinct vision-grounded environments.
• Analyzing model performance on strategic reasoning tasks in multi-agent settings.

Strengths

• Evaluates fourteen state-of-the-art Vision-Language Models.
• Covers eight distinct vision-grounded environments.
• Employs two complementary evaluation dimensions: offline strategic reasoning and online decision-making.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

huggingface

Freshness

Last updated 2025-06-04 02:03:09; freshness should be verified.