Ayn-VQA: Culturally Grounded Arabic Vision-Language Evaluation Dataset

Ayn-VQA is a culturally grounded Arabic multimodal evaluation dataset designed for the ImageEval 2026 Shared Task at ArabicNLP 2026. It tests whether a model can read a culturally specific image from a spoken Arabic question and distinguish grounded descriptions from plausible hallucinations. The dataset is authored by QCRI and was last updated in June 2026.

Use Cases

• Benchmarking multimodal model performance on culturally specific Arabic images based on the described evaluation task
• Evaluating model ability to answer spoken Arabic questions about images based on the dataset's language tracks
• Testing model robustness against hallucinated descriptions based on the task of telling grounded descriptions apart from plausible but hallucinated ones

Strengths

• Dataset is explicitly designed for a major shared task (ImageEval 2026 at ArabicNLP 2026), suggesting a defined evaluation purpose
• Offers two separate language tracks (English and Modern Standard Arabic) for scoring, enabling cross-lingual analysis
• Focuses on culturally specific Arabic images, addressing a potential gap in multimodal evaluation

Limitations

• Description metadata is limited; actual data quality requires manual inspection after download
• Column-level documentation is absent; field semantics must be inferred after download
• Row count is unknown, which may limit suitability assessment

Provenance

Source

QCRI

Freshness

Last updated 2026-06-08 22:29:39; freshness should be verified