Autoresearch Novelty Bench: AI Agent Hypothesis Benchmark

A benchmark dataset for testing whether autonomous AI research agents propose novel, mechanism-distinct hypotheses. It contains 10,380 rows of experimental training runs built on Prime Intellect's autonomous-speedrunning archive. The dataset was created by Evo and last updated on May 17, 2026.

Use Cases

• Benchmarking AI research agent novelty based on the described experimental training runs.
• Evaluating hypothesis generation distinctness using the described mechanism-distinct criteria.
• Analyzing autonomous agent performance in optimization speedruns as described in the source archive.

Strengths

• 10,380 rows of experimental training runs provide a substantial testbed.
• Built on a specific, documented archive from Prime Intellect's autonomous-speedrunning project.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count beyond the 10,380 experiments is unknown, which may limit suitability assessment.

Provenance

Source

Prime Intellect's autonomous-speedrunning archive.

Collection Method

Built from two AI agents (Claude Code and Codex) competing on modded-nanogpt's optimization speedrun.

Freshness

Last updated 2026-05-17 17:09:32.