The Heap: A Contamination-Free Multilingual Code Dataset for 57 Languages

57 programming languages are represented in The Heap, a dataset designed for reproducible evaluation of large language models. It was developed by AISE-TUDelft and last updated in May 2026. The dataset is described as contamination-free, aiming to prevent data leakage between training and evaluation sets.

Use Cases

• Benchmarking LLM code generation performance based on its multilingual composition.
• Studying cross-lingual transfer in code models based on the 57-language structure.
• Reproducing LLM evaluation results based on the dataset's contamination-free design.

Strengths

• Covers 57 distinct programming languages.
• Explicitly designed to be contamination-free for evaluation reproducibility.
• Includes a reproduction package for validation.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count and total dataset size are unknown, which may limit suitability assessment.

Provenance

Source

AISE-TUDelft

Freshness

Last updated 2026-05-13 12:06:13; freshness should be verified.