Ranked Science Questions for Language Model Alignment

ARC-Challenge is a dataset testing grade-school level science understanding, widely adopted for evaluating LLM reasoning. AbacusAI created a paired preference-ranked version from its train split, focusing on the more difficult 'Challenge' subsection. The dataset was last updated in February 2024.

Use Cases

• Fine-tuning language models via Direct Preference Optimization using paired question-response rankings.
• Benchmarking LLM reasoning on specific science question prompts (x) from the ARC-Challenge subset.
• Analyzing model performance on the 'Challenge' subsection to measure world understanding.
• Creating few-shot learning prompts from the ranked preference data for science QA tasks.

Strengths

• Derived from the widely adopted ARC-Challenge benchmark for LLM evaluation.
• Focuses on the more difficult 'Challenge' subsection of the ARC dataset.
• Structured as a paired preference dataset suitable for alignment training.

Limitations

• Specific row count, column names, and sample size are not provided.
• Limited to the train split of ARC-Challenge, excluding other data splits.
• Potential class or topic imbalance inherent to the original ARC question distribution.

Provenance

Source

ARC (AI2 Reasoning Challenge) dataset, specifically the 'Challenge' subsection.

Collection Method

Created from the train split of ARC-Challenge, transformed into paired preference-ranked format.

Freshness

Last updated 2024-02-26.