GitHub Code Files in 32 Programming Languages

Encompassing 115 million code files from GitHub, totaling 1 TB of text data. It includes code in 32 programming languages across 60 file extensions and was created from the GitHub dataset on BigQuery by codeparrot.

Use Cases

• Train a language model for code generation using the 115 million code files across 32 programming languages.
• Analyze the distribution and frequency of code across 60 different file extensions.
• Perform multilingual code analysis by leveraging the dataset's coverage of 32 distinct programming languages.

Strengths

• Contains 115 million individual code files, providing a large-scale corpus.
• Covers 32 distinct programming languages, offering multilingual breadth.
• Totals 1 TB of raw text data for substantial model training.

Limitations

• The specific columns and schema are unknown, limiting structured analysis.
• The dataset's last update was in October 2022, so it may not include recent code trends or languages.
• The license is unspecified, which could restrict commercial or research use.

Provenance

Source

GitHub dataset on BigQuery.

Collection Method

Created from the GitHub dataset on BigQuery.

Freshness

Last updated 2022-10-20.