PRDbench Code Agent Evaluation Test Cases

PRDbench is a benchmark dataset with 50 test cases for evaluating code agents' development capabilities in real-world environments. Each test case includes a PRD requirement (PRD query) and an acceptance scoring scheme (Criteria). The dataset is authored by AGI-Eval and was last updated in December 2025.

Use Cases

• Evaluate code agents' performance by scoring their outputs against the provided acceptance Criteria for each PRD query.
• Benchmark the ability of code agents to interpret and implement requirements from PRD query specifications.
• Analyze the correlation between different acceptance Criteria and agent success rates across the 50 test cases.

Strengths

• Contains 50 specifically designed test cases for structured evaluation.
• Each test case includes both a PRD query and a defined acceptance scoring scheme (Criteria).
• Designed for real-world scenario evaluation of code development capabilities.

Limitations

• Small scale with only 50 test cases, limiting statistical power for broad conclusions.
• Unknown data structure, column details, and sample data availability.
• Potential for narrow scope focused on specific types of PRD requirements.

Provenance

Source

AGI-Eval on Hugging Face.

Collection Method

Designed as a benchmark for evaluating code agents.

Freshness

Last updated on 2025-12-05.