SciGenBench: Scientific Image Synthesis Benchmark

SciGenBench is a benchmark dataset for evaluating scientific image generation, focusing on logical correctness beyond visual plausibility. It was created by author J017athan and last updated in January 2026. The dataset employs a hybrid evaluation protocol combining Inverse Quiz Validation and LMM-as-Judge methodologies.

Use Cases

• Benchmarking text-to-image models on logical correctness using the Inverse Quiz Validation (Rinv) metric.
• Evaluating multimodal reasoning capabilities of models via the LMM-as-Judge protocol on generated scientific diagrams.
• Training or fine-tuning models for verifiable diagram generation using the Understand → Plan → Code framework of ImgCoder.
• Studying the downstream utility of high-fidelity synthetic images for strengthening multimodal reasoning tasks.

Strengths

• Employs a hybrid, multi-method evaluation protocol (Inverse Quiz Validation + LMM-as-Judge) for robust assessment.
• Focuses on the critical dimension of logical correctness for scientific imagery, beyond standard visual metrics.

Limitations

• Specific dataset size, row count, and column details are unknown.
• The geographic scope and temporal coverage of the source material are unspecified.

Provenance

Source

huggingface

Collection Method

Benchmark dataset created for systematic study of scientific image generation; methodology involves synthetic image generation and multimodal evaluation.

Freshness

Last updated on 2026-01-27.