Python Code Repository Benchmarks from GitHub

RepoBench v1.1 (Python) is a dataset for evaluating code generation models, derived from GitHub repositories. The collection spans from October 6th to December 31st, 2023, and has been deduplicated against the Stack v2 dataset to prevent data leakage. It was created by author 'tianyang' for the ICLR 2024 conference.

Use Cases

• Benchmarking model performance on Python code generation tasks using repository-level context.
• Training models to generate or complete Python functions based on surrounding file and repository structure.
• Studying patterns in real-world Python code from a specific 3-month period in late 2023.
• Evaluating model memorization and generalization by testing on deduplicated code samples.

Strengths

• Data is deduplicated against a major code dataset (Stack v2) to mitigate memorization concerns.
• Collection is temporally focused on a specific 3-month period from late 2023.

Limitations

• The exact number of rows, files, or repositories is unknown.
• Limited to Python code from a narrow 3-month window, which may not represent long-term coding trends.
• Potential geographic or project bias inherent in the GitHub source.

Provenance

Source

GitHub repositories.

Collection Method

Collected and deduplicated against the Stack v2 dataset.

Time Range

October 6th to December 31st, 2023.

Freshness

Last updated on the Hugging Face platform on February 27, 2024.