Japanese VLM Benchmark Collection with Human-Refined Annotations

JAMMEval is a curated benchmark collection for evaluating Vision-Language Models on Japanese Visual Question Answering tasks. It refines seven existing Japanese VQA evaluation datasets through two rounds of human annotation to improve reliability. The dataset was created by llm-jp and was last updated in April 2026.

Use Cases

• Benchmarking VLM performance on Japanese VQA tasks using human-refined question-answer pairs.
• Evaluating model robustness by analyzing performance across seven refined constituent datasets.
• Training or fine-tuning VLMs on high-quality Japanese image-text pairs from the curated collection.

Strengths

• Comprises seven refined Japanese VQA evaluation datasets.
• Underwent two rounds of human annotation to improve data quality.

Limitations

• Specific sample sizes, row counts, and data formats for the constituent datasets are not provided.
• The geographic and temporal coverage of the source images and questions is unspecified.

Provenance

Source

Refinement of seven existing Japanese VQA evaluation datasets.

Collection Method

Constructed through two rounds of human annotation.

Freshness

Last updated in April 2026.

Geography

Japan (implied by focus on Japanese language tasks).