BigCodeBench Programming Evaluation Suite

1140 programming tasks are available in two variants: one for code completion from docstrings and another for code generation from natural language instructions. Each task includes an average of 5.6 test cases with 99% code coverage. The dataset was created by the BigCode project and updated in April 2025.

Use Cases

• Benchmarking model performance on code completion using structured docstrings as prompts.
• Evaluating code generation models on tasks defined by natural language instructions.
• Analyzing the relationship between prompt characteristics (e.g., average 1112.5 characters for 'Complete' variant) and solution quality.
• Studying test case coverage and correctness for generated code solutions.

Strengths

• Contains 1140 distinct programming tasks.
• Each task is supported by an average of 5.6 test cases with 99% coverage.
• Provides two distinct task variants for different evaluation scenarios.

Limitations

• Specific row count, column names, and data size are not provided.
• The dataset focuses on benchmark tasks rather than a large-scale training corpus.
• Limited information on the diversity of programming languages or problem domains covered.

Provenance

Source

bigcode project on Hugging Face.

Collection Method

Expert-generated programming tasks and test cases.

Freshness

Last updated on 2025-04-30.