CritPt: Frontier Physics Research Benchmark with <1,000 Reasoning Tasks

CritPt contains fewer than 1,000 research-level physics reasoning tasks covering domains such as condensed matter, quantum physics, and astrophysics. Developed by the CritPt-Benchmark team and released in late 2024, it serves as a frontier benchmark for testing LLM capabilities on unpublished scientific problems.

Use Cases

• Evaluating LLM reasoning accuracy using the physics problem text
• Benchmarking model performance across specific sub-domains like astrophysics or quantum physics
• Testing multi-step logical deduction in frontier scientific research contexts

Strengths

• Focuses on unpublished research-level tasks to prevent LLM data contamination
• Covers diverse physics sub-fields including AMO and condensed matter
• Released under Apache 2.0 license for open research use

Limitations

• Small sample size of fewer than 1,000 records
• High domain specificity requires expert-level knowledge for manual output verification

Provenance

Source

CritPt-Benchmark (Arxiv 2509.26574)

Collection Method

Expert-curated research-level physics tasks

Freshness

Last updated November 2025.