Chemical Reaction Visual Question Answering Benchmark With 1,525 Questions

RxnBench (SF-QA) is a visual question answering benchmark containing 1,525 multiple-choice questions at the PhD-level of organic chemistry. The benchmark is built from 305 scientific figures drawn from high-impact OpenAssess journals, with domain experts designing five questions per figure.

Use Cases

• Train multimodal models on the 1,525 PhD-level multiple-choice questions to test visual understanding of chemical reaction figures.
• Benchmark model performance on interpreting scientific figures from high-impact OpenAssess journals.
• Develop specialized visual question answering systems for the organic chemistry domain using the expert-designed question set.

Strengths

• Contains 1,525 expert-designed, PhD-level multiple-choice questions for rigorous evaluation.
• Built from 305 scientific figures sourced from high-impact OpenAssess journals.
• Focuses on the specific, complex domain of organic chemistry reaction understanding.

Limitations

• Limited to 305 source figures, which may restrict the diversity of visual reaction representations.
• The PhD-level focus makes the dataset highly specialized and potentially less accessible for general-purpose models.
• Sample data and column structure are not provided in the input, complicating initial data exploration.

Provenance

Source

UniParser

Collection Method

Built from 305 scientific figures drawn from high-impact OpenAssess journals, with domain experts designing five multiple-choice questions per figure.

Freshness

Last updated on 2026-03-05.