Counting Benchmark for Vision Language Models

491 images from the CountBench benchmark evaluate object counting in vision-language models. The dataset was automatically curated and manually verified from the LAION-400M dataset, introduced by author vikhyatk for the PaliGemma model.

Use Cases

• Evaluate object counting performance of vision-language models on 491 manually verified images.
• Benchmark model accuracy on counting tasks derived from the LAION-400M image-text dataset.
• Assess the robustness of multimodal AI systems using a curated set of images containing objects to count.

Strengths

• 491 manually verified images provide a focused evaluation set.
• Derived from the large-scale LAION-400M dataset, ensuring a diverse origin.
• Specifically designed for benchmarking counting in vision-language models like PaliGemma.

Limitations

• Smaller than the original CountBench, containing only 491 images instead of 540 due to broken URLs.
• Limited to a single, specific task of object counting evaluation.
• No column or sample data information is available for detailed inspection.

Provenance

Source

Curated from the LAION-400M image-text dataset.

Collection Method

Automatically curated and manually verified.

Freshness

Last updated on October 22, 2024.