Dolmino Mix 1124: High-Quality Text Mixture for OLMo2 Training

Dolmino Mix 1124 is a mixture of high-quality data used for the second stage of OLMo2 training. The dataset aggregates text from multiple sources, including 752 billion tokens from DCLM web pages, 58.6 billion tokens from Pes2o STEM papers, and 1.26 billion tokens from StackExchange code text. It was created by AllenAI and last updated on October 29, 2025.

Use Cases

• Training large language models based on the described mixture of high-quality web, academic, and code text.
• Fine-tuning language models for STEM-related tasks based on the inclusion of 58.6 billion tokens from STEM papers.
• Developing code generation or understanding models based on the inclusion of 1.26 billion tokens from StackExchange.
• Benchmarking text quality and diversity in pre-training corpora based on the multi-source mixture.

Strengths

• Aggregates 752 billion tokens from the DCLM source of high-quality web pages.
• Includes 58.6 billion tokens from the Pes2o source of STEM academic papers.
• Contains 1.26 billion tokens from the StackExchange source of code-related text.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count and total dataset size are unknown, which may limit suitability assessment.
• Data may reflect source bias inherent to the specific web, academic, and forum sources used.

Provenance

Source

AllenAI, aggregated from multiple sources including DCLM, Flan, Pes2o, Wikipedia, and StackExchange.

Collection Method

Likely aggregated and filtered from the listed open data sources for language model training.

Freshness

Last updated 2025-10-29 17:55:04.