Code-LMs: Test Sets for 12 Programming Languages to Evaluate Code Models

Frank F. Xu from Carnegie Mellon University created datasets for the paper "A Systematic Evaluation of Large Language Models of Code". The data includes test sets of approximately 100 files each in 12 programming languages, which were not included in The Pile training corpus. These test sets were used to evaluate models including Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, CodeParrot, and the authors' PolyCoder model.

Use Cases

• Benchmarking code generation models based on unseen test sets in multiple programming languages.
• Evaluating model performance on code not present in common training corpora like The Pile.
• Comparing proprietary models like Codex against open-source models like GPT-J and PolyCoder.

Strengths

• Test sets contain approximately 100 files for each of 12 distinct programming languages.
• Files are explicitly not part of The Pile, providing a clean evaluation for models trained on that corpus.
• Includes an index of training set file paths and commit SHAs for reproducibility.

Limitations

• Row count and total dataset size are unknown, which may limit suitability assessment.
• Column-level documentation is absent; field semantics must be inferred after download.
• Last update date is unknown; freshness unverified.

Provenance

Source

Carnegie Mellon University

Collection Method

Created for the systematic evaluation of large language models of code, as described in the associated arXiv paper.