The Stack V2 Dedup: Near-Deduplicated Source Code from 600+ Languages

The Stack V2 Dedup is a near-deduplicated collection of source code containing between 1 billion and 10 billion records across 600+ programming languages. Produced by BigCode and last updated in April 2024, it serves as a refined subset of the full Stack v2 dataset for training large language models.

Use Cases

• Training code-specific LLMs using the multilingual 'text' modality
• Benchmarking deduplication efficiency across 600+ programming languages
• Repository-level code analysis using the repository-grouped data organization

Strengths

• Includes 600+ programming languages
• Near-deduplicated to minimize data leakage and training bias
• Scale of 1 billion to 10 billion records

Limitations

• Custom license ('Licenseother') may restrict commercial usage or redistribution
• Massive scale (1B+ records) requires significant storage and high-performance computing resources

Provenance

Source

BigCode

Collection Method

Scraped from public software repositories

Freshness

Last updated April 2024.

Geography

Global