BioMysteryBench Preview: A 5-Problem Sample for AI Benchmarking

A public sample containing 5 problems from the BioMysteryBench benchmark created by Anthropic. The dataset includes problem prompts, grading rubrics, and human solvability indicators. The preview was last updated on 2026-04-29.

Use Cases

• Benchmarking AI model performance on biology-related reasoning tasks based on the provided problem prompts.
• Analyzing the difficulty of problems for AI versus humans based on the human_solvable flag.
• Testing AI agents in constrained web environments based on the allowed network domains specification.
• Developing or refining evaluation rubrics for open-ended questions based on the provided answer_rubric field.

Strengths

• Includes explicit grading criteria (answer_rubric) for each problem.
• Contains a human_solvable indicator, allowing comparison between human and AI performance.
• Provides constraints on the solving environment (allowed_domains) for realistic agent testing.

Limitations

• Description metadata is limited; actual data quality requires manual inspection after download.
• Row count is unknown, which may limit suitability assessment.
• Column-level documentation is absent; field semantics must be inferred after download.

Provenance

Source

Anthropic

Collection Method

Likely created as a benchmark for evaluating AI models.

Freshness

Last updated 2026-04-29 03:32:33; freshness should be verified.