AI2 ARC: 7,787 Grade-School Science Multiple-Choice Questions

AI2 ARC consists of 7,787 genuine grade-school science questions compiled by the Allen Institute for AI in 2018. The dataset is partitioned into Easy and Challenge sets, with the latter specifically containing questions that simple retrieval-based and word co-occurrence algorithms fail to solve. It is accompanied by a supporting corpus of 14 million science-related sentences for open-domain question answering research.

Use Cases

• Benchmarking reasoning in LLMs using the Challenge Set partition
• Training retrieval-augmented generation (RAG) systems using the 14-million sentence science corpus
• Evaluating multiple-choice question answering performance on academic science content

Strengths

• 7,787 genuine grade-school science questions
• Challenge Set specifically filters out questions solvable by simple retrieval algorithms
• Includes a 14-million sentence science corpus for retrieval-augmented generation

Limitations

• Monolingual English focus
• Domain-specific to science curriculum
• Relatively small question count of 7,787 records compared to modern benchmarks

Provenance

Source

Allen Institute for AI (AI2)

Collection Method

Assembled from genuine grade-school science questions

Time Range

2018

Freshness

Last updated in December 2023; core question set originates from 2018.