SWE-bench Lite: 300 GitHub Issue-Pull Request Pairs for Automated Code Repair

300 test Issue-Pull Request pairs from 11 popular Python repositories, released as part of the SWE-bench research project. The dataset was created to test systems' ability to automatically resolve real-world GitHub issues, with evaluation performed by verifying unit tests using post-PR behavior as the reference solution. It was released by the SWE-bench organization and last updated on 2025-04-29.

Use Cases

• Benchmarking language models for automated GitHub issue resolution based on the described Issue-PR pairs.
• Training or evaluating code generation agents on real-world software engineering problems based on the 300 test cases.
• Studying the effectiveness of unit test verification as an evaluation metric for code changes.

Strengths

• Contains 300 concrete test cases for evaluation.
• Focuses on real-world issues from 11 popular Python repositories.
• Evaluation is based on unit test verification, providing an objective metric.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

SWE-bench organization via Hugging Face.

Collection Method

Collected as a subset of the SWE-bench dataset, gathering Issue-Pull Request pairs from GitHub.

Freshness

Last updated 2025-04-29 20:49:19; freshness should be verified.