Lbpp: 162 Fresh Python Programming Problems with Unit Tests

162 programming problems were created by CohereLabs to be fresh and more difficult than benchmarks like HumanEval and MBPP. The dataset was last updated on June 9, 2025. It is structured as a drop-in replacement for similar code generation benchmarks.

Use Cases

• Benchmarking code generation models based on the described collection of 162 problems.
• Training models for Python programming tasks based on the described unit-tested examples.
• Evaluating model performance on 'fresh' problems not leaked to training data.
• Augmenting existing code datasets like HumanEval with more difficult examples.

Strengths

• Contains 162 distinct programming problems.
• Designed to be more difficult than similar benchmark datasets.
• Includes accompanying unit tests for each problem.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Freshness should be verified as the last update was in 2025.

Provenance

Source

CohereLabs

Collection Method

Created as a collection of programming problems.

Freshness

Last updated 2025-06-09.