PeerReview Bench: Expert-Annotated Scientific Paper Reviews for AI Evaluation

PeerReview Bench is a dataset of expert-annotated review items from scientific papers, organized for three complementary evaluation tasks. The data is intended for evaluation, not training, and was created by prometheus-eval. The dataset page was last updated on 2026-05-27.

Use Cases

• Benchmarking AI models on scientific paper review tasks based on expert annotations.
• Evaluating the quality of AI-generated peer reviews based on structured review items.
• Training or testing systems for automated paper critique based on annotated review points.

Strengths

• Data is expert-annotated, suggesting a high-quality benchmark.
• Designed for three complementary evaluation tasks, providing structured assessment scenarios.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Last updated 2026-05-27 21:14:49; freshness should be verified.

Provenance

Source

prometheus-eval

Collection Method

Likely collected and annotated by experts as part of the CMU Paper Reviewer project.

Freshness

2026-05-27