Multimodal Hallucination Benchmark for Visual Question Answering

96 challenging questions based on images from OpenImages form this evaluation benchmark for hallucination in Large Multimodal Models. It includes ground-truth answers and image contents. The dataset was created by Shengcao1006 and uploaded in November 2023.

Use Cases

• Benchmark model performance on hallucination using the 96 question-answer pairs and corresponding images.
• Analyze failure modes in LMMs by comparing generated answers against the provided ground-truth answers.
• Train or fine-tune models to reduce hallucination using the curated image content descriptions and questions.
• Develop new evaluation metrics for multimodal hallucination based on the benchmark's structured challenges.

Strengths

• 96 specifically designed challenging questions for targeted evaluation.
• Includes ground-truth answers and image content for precise scoring.

Limitations

• Small scale with only 96 data points, limiting statistical power for broad conclusions.
• Image source is limited to OpenImages, which may not represent all visual domains.

Provenance

Source

Hugging Face, created by Shengcao1006.

Collection Method

Curated benchmark; questions based on images from OpenImages.

Time Range

null

Freshness

Last updated in November 2023.

Geography

null