CRYSTAL: Diagnostic Benchmark for Multimodal Step-by-Step Reasoning

CRYSTAL is a diagnostic benchmark for evaluating multimodal reasoning step by step, not just by the final answer. Each instance pairs an image and a question with an ordered sequence of natural-language reference reasoning steps, enabling step-level metrics like Match F1 and Ordered Match F1 alongside answer accuracy. The dataset was created by author waybarrios and was last updated on the platform in June 2026.

Use Cases

• Benchmarking model reasoning traceability based on step-level metrics like Match F1
• Evaluating the logical coherence of multimodal AI based on ordered sequences of reasoning steps
• Training models for intermediate inference generation based on image-question pairs with reference steps
• Diagnosing failure points in multimodal reasoning pipelines based on step-by-step evaluation

Strengths

• Provides step-level diagnostic metrics (Match F1 and Ordered Match F1) beyond final answer accuracy
• Includes ordered sequences of natural-language reference reasoning steps for each image-question pair
• Designed specifically for evaluating multimodal reasoning processes, not just outputs

Limitations

• Row count, file formats, and column-level documentation are unknown, limiting suitability assessment
• The full description is truncated, requiring a visit to the dataset page for complete details
• License information is unknown, which may restrict usage

Provenance

Source

huggingface

Collection Method

Likely created as a research benchmark; specific gathering method is not detailed in the provided description.

Freshness

Last updated 2026-06-24 16:35:02