ClimbLab: A 1.2-Trillion-Token Pre-Training Corpus with 20 Clusters

NVIDIA's ClimbLab is a 1.2-trillion-token corpus for language model pre-training. It was created by OptimalScale using a semantic clustering method called CLIMB to reorganize and filter data from the Nemotron-CC and SmolLM-Corpus sources into 20 distinct clusters. The dataset was last updated on Hugging Face in May 2025.

Use Cases

• Pre-training large language models based on the described 1.2-trillion-token high-quality corpus.
• Evaluating clustering algorithms for text data based on the described CLIMB-clustering methodology.
• Studying the impact of topic-based semantic grouping on model performance based on the described 1,000 initial topic groups.
• Creating specialized language models for domains identified within the 20 distinct clusters.

Strengths

• Corpus contains 1.2 trillion tokens, indicating a substantial scale for training.
• Data is organized into 20 distinct semantic clusters, suggesting a structured and potentially more focused resource.
• Derived from established sources Nemotron-CC and SmolLM-Corpus, implying a foundation on known datasets.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment for specific tasks.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

OptimalScale, derived from Nemotron-CC and SmolLM-Corpus.

Collection Method

Filtered and semantically reorganized using CLIMB-clustering.

Freshness

Last updated 2025-05-04 03:48:49; freshness should be verified.