Code Bagel: Multilingual Source Code Corpus with 800 Million Tokens

Third-Space's Code Bagel is an unofficial reupload of a dataset containing source code. The description indicates it contains around 800 million tokens of unique coding data, supporting over 100 programming languages. The dataset was last updated on the Hugging Face platform on October 8, 2024.

Use Cases

• Train code generation models based on the multilingual source code corpus.
• Fine-tune models for specific programming languages based on the dataset's language support.
• Conduct analysis on coding patterns and token distributions across different languages.
• Create benchmarks for code-related NLP tasks using the described volume of data.

Strengths

• Contains approximately 800 million tokens of unique coding data.
• Supports source code in over 100 different programming languages.
• Lines are configured with a maximum length of 10,000 tokens.

Limitations

• Description metadata is limited; actual data quality requires manual inspection after download.
• Column-level documentation is absent; field semantics must be inferred after download.
• Row count, file formats, and license information are unknown.

Provenance

Source

Reupload of the original 'rombodawg/code_bagel' dataset from Hugging Face.

Collection Method

Likely aggregated from various public code repositories, though the exact method is unspecified.

Freshness

Last updated 2024-10-08 22:42:41; freshness should be verified.