BenchLMM: Cross-Style Visual Capability Benchmark for LMMs

BenchLMM evaluates the cross-style visual reasoning capabilities of Large Multimodal Models (LMMs) across diverse image styles. Developed by AIFEG and presented at ECCV 2024, this benchmark assesses how models generalize beyond standard natural images to various artistic and synthetic domains.

Use Cases

• Benchmarking Large Multimodal Models (LMMs) on non-natural image styles
• Analyzing model robustness to visual style shifts using the provided cross-style image sets
• Evaluating zero-shot visual reasoning across artistic or synthetic domains

Strengths

• Peer-reviewed at ECCV 2024
• Focuses on cross-style visual generalization beyond natural images
• Released under the permissive Apache-2.0 license

Limitations

• Primarily designed for evaluation rather than large-scale model training
• Lack of detailed metadata regarding the specific number of images per style

Provenance

Source

AIFEG, ECCV 2024

Freshness

Last updated August 2024 following its ECCV 2024 publication.