SWE-bench Verified: 500 Human-Validated GitHub Issue-PR Pairs

500 human-validated GitHub Issue-Pull Request pairs from popular Python repositories, curated by the SWE-bench team. This subset of the original benchmark focuses on high-quality samples verified for evaluation accuracy through manual review. It serves as a rigorous test for autonomous systems attempting to solve real-world software engineering tasks.

Use Cases

• Evaluating autonomous agents on resolving GitHub issues using the issue description and PR pairs
• Benchmarking code generation models against unit test verification suites
• Analyzing software engineering task success rates across different Python repositories

Strengths

• 500 human-validated samples ensuring high-quality ground truth
• Unit test verification using post-PR behavior as a reference
• Sourced from popular, real-world Python repositories

Limitations

• Small sample size of 500 records compared to the full benchmark
• Restricted exclusively to the Python programming language
• Snapshot-based data may not reflect the current state of the source repositories

Provenance

Source

SWE-bench

Collection Method

Human-validated subset of scraped GitHub Issue-Pull Request pairs

Freshness

Last updated February 2026.