SoundnessBench: 1,099 ICLR Research Proposals with Reviewer Soundness Scores

1,099 machine learning research proposals reconstructed from ICLR submissions and labeled with reviewer soundness sub-scores. The benchmark, created by hosytuyen, is intended to evaluate whether LLMs can judge proposal-stage methodological soundness before expensive experimentation. It was last updated on May 30, 2026.

Use Cases

• Benchmarking LLMs on scientific peer-review tasks based on labeled soundness scores.
• Training models to predict methodological soundness based on proposal-stage text.
• Analyzing the relationship between proposal content and reviewer judgments.
• Developing tools for pre-experimentation research proposal screening.

Strengths

• Contains 1,099 labeled research proposals, providing a substantial corpus for evaluation.
• Data is derived from a specific, credible source: submissions to the ICLR conference.
• Labels include reviewer soundness sub-scores, enabling fine-grained analysis.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is known, but the specific features and structure of each proposal are not detailed.
• Data may reflect temporal and topical bias inherent to its source conference.

Provenance

Source

Reconstructed from ICLR (International Conference on Learning Representations) submissions.

Collection Method

Reconstructed and labeled with reviewer soundness sub-scores.

Freshness

Last updated 2026-05-30 08:22:41; freshness should be verified.