Collider-Bench is an AI benchmark for evaluating LLM agents on their ability to reproduce experimental analyses from the Large Hadron Collider at CERN. The benchmark uses public papers and open scientific software to test multi-step scientific reasoning by autonomous coding agents. The dataset was authored by Dariusfar and last updated on 2026-05-13.
Use Cases
- Benchmarking LLM agents on multi-step scientific reasoning based on tasks described in the dataset.
- Evaluating AI's ability to read published CMS or ATLAS searches and identify relevant signal regions.
- Testing autonomous agents on generating and processing simulated signal events for LHC analyses.
- Assessing AI performance on implementing event selection and predicting binned signals from scientific papers.
Strengths
- Specifically designed for evaluating autonomous coding agents on complex, multi-step scientific tasks.
- Based on real-world public papers and open scientific software from CERN's LHC experiments.
- Last updated on 2026-05-13, indicating recent maintenance.
Limitations
- Column-level documentation is absent; field semantics must be inferred after download.
- Row count is unknown, which may limit suitability assessment.
- Description metadata is limited; actual data quality requires manual inspection after download.
Provenance
- Source
- Dariusfar
- Collection Method
- Likely compiled from public CMS and ATLAS papers and associated open scientific software.
- Freshness
- Last updated 2026-05-13 04:28:32; freshness should be verified.