Multilingual Code Samples from 30 Programming Languages

2.6GB of text comprises a 0.1% subset of the-stack dataset, containing 10,000 random samples for each of 30 programming languages. The dataset was created by bigcode and released in May 2023.

Use Cases

• Fine-tune a language model for code completion using the 'python' language samples.
• Train a classifier to identify programming languages from raw source code text across 30 language categories.
• Analyze coding style and syntax patterns within specific languages like 'java' or 'rust'.
• Benchmark model performance on multilingual code understanding tasks using the balanced 10,000-sample-per-language structure.

Strengths

• Contains 10,000 samples for each of 30 programming languages, ensuring balanced representation per language.
• Total dataset size is 2.6GB of raw text (code), providing substantial material for model training.

Limitations

• The dataset is a small (0.1%) random subset of a larger corpus, which may limit coverage of rare code patterns.
• Specific row counts, column definitions, and data schema details are not provided in the available description.

Provenance

Source

bigcode on Hugging Face, derived from the-stack dataset.

Collection Method

Random sampling of 10,000 code files per language from the original the-stack dataset.

Freshness

Last updated on 2023-05-02.