Real-World Visual Question Answering Benchmark from Vehicle Images

Over 700 anonymized images, primarily captured from vehicles, form this multimodal benchmark. Each image is paired with a question and a verifiable answer, designed to test real-world scene understanding. The dataset was released by xAI in April 2024.

Use Cases

• Benchmarking model performance on visual question answering using the provided image-question-answer triplets.
• Training models to answer questions about real-world objects and scenes captured in the 700+ vehicle-centric images.
• Evaluating the robustness of vision models on anonymized, real-world imagery distinct from curated web photos.
• Analyzing model failures on 'easily verifiable' visual questions to improve reasoning capabilities.

Strengths

• Over 700 real-world images provide a testbed for visual question answering.
• Each data point includes a question and a verifiable answer for structured evaluation.

Limitations

• The dataset size of 700 images is relatively small for training large models from scratch.
• The focus on vehicle-captured and anonymized images may introduce a geographic and perspective bias.

Provenance

Source

xAI (xai-org).

Collection Method

Anonymized images captured from vehicles and other real-world sources.

Freshness

Last updated in April 2024.