GitHub Issue and Pull Request Pairs from 12 Python Repositories

2,294 Issue-Pull Request pairs test automated resolution of real-world software problems. The dataset was created by the SWE-bench project to evaluate systems using unit test verification against post-PR behavior. It is sourced from 12 popular Python repositories.

Use Cases

• Evaluate code generation models on their ability to resolve real-world GitHub issues using the provided problem_statements.
• Benchmark automated software engineering tools against the 2,294 Issue-Pull Request pairs for unit test verification.
• Analyze patterns in successful pull requests that solve specific types of software issues documented in the dataset.

Strengths

• Contains 2,294 real-world software engineering problems with verified solutions.
• Sourced from 12 popular and actively maintained Python repositories, ensuring practical relevance.
• Designed for rigorous evaluation using unit test verification against post-PR behavior.

Limitations

• Limited to 12 Python repositories, which may not represent the diversity of software engineering issues in other languages or ecosystems.
• The dataset focuses on problem statements, potentially omitting intermediate code states or development context.
• Evaluation is tied to specific unit tests, which may not capture all aspects of a correct solution.

Provenance

Source

GitHub issues and pull requests from 12 popular Python repositories.

Collection Method

Collected 2,294 Issue-Pull Request pairs for evaluation.

Freshness

Last updated on 2025-04-29.