The Stack: 3TB of Permissively Licensed Source Code in 30+ Languages

BigCode released The Stack in late 2022, a 3TB collection of source code spanning 30+ programming languages and 193 permissive licenses. The dataset contains between 100 million and 1 billion near-deduplicated records of public code files scraped from the web.

Use Cases

• Training large language models for code generation using the source file content
• Analyzing software licensing distribution across 193 permissive types
• Benchmarking deduplication algorithms on a 3TB text corpus

Strengths

• 3TB near-deduplicated volume
• 193 permissive license categories
• Coverage of 30+ programming languages

Limitations

• Exclusion of copyleft codebases reduces diversity of software patterns
• Temporal staleness as of April 2023
• Potential for sensitive information or PII within source code comments

Provenance

Source

BigCode (Service of Hugging Face and ServiceNow)

Collection Method

Scraped from public code repositories

Freshness

Last updated April 2023; does not include code changes or new repositories created after this date.

Geography

Global