OpenAI HumanEval: 164 Handwritten Python Programming Problems

164 handwritten Python programming problems were released by OpenAI in 2021 to evaluate the functional correctness of code generation models. Each entry provides a function signature, docstring, reference implementation, and unit tests for automated validation.

Use Cases

• Evaluating code generation models using the 'docstring' and 'function signature' as prompts
• Benchmarking functional correctness by executing generated code against the provided 'unit tests'
• Fine-tuning small language models on high-quality Python implementations found in the 'body' field

Strengths

• 164 expert-handwritten problems
• Includes unit tests for automated verification
• MIT licensed

Limitations

• Small sample size of 164 records
• Restricted to Python syntax
• Susceptible to benchmark contamination in models trained on post-2021 web data

Provenance

Source

OpenAI (Arxiv 2107.03374)

Collection Method

expert-generated

Time Range

2021

Freshness

Static benchmark released in 2021; last metadata update in 2024.