OLMo Mix 1124: OLMo-2 Language Model Pretraining Corpus

OLMo Mix 1124 is a collection of data used to train the OLMo-2-1124 models, released in November 2024. The majority of the dataset, 3.70 trillion tokens, comes from the DCLM-Baseline source. It was created by AllenAI and includes components such as ArXiv papers, pes2o, StarCoder, and Algebraic-stack.

Use Cases

• Pretraining large language models based on the multi-trillion token text corpus.
• Analyzing the impact of different data sources (e.g., scientific papers, code) on model performance.
• Studying dataset composition and licensing for responsible AI development.
• Benchmarking language model capabilities on mixed-domain text.

Strengths

• Contains 3.70 trillion tokens from the DCLM-Baseline source, providing substantial scale for pretraining.
• Includes a documented breakdown of component sources like ArXiv (20.8B tokens) and StarCoder (83.0B tokens).
• Explicitly states licenses for major components, such as CC-BY-4.0 for DCLM-Baseline.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• The description is incomplete, referencing a full breakdown on an external page.

Provenance

Source

AllenAI, compiled from multiple sources including DCLM-Baseline, ArXiv, pes2o, and StarCoder.

Collection Method

Collection of data used for model pretraining, with the majority sourced directly from DCLM-Baseline without additional filtering.

Time Range

Associated with the OLMo-2 model release in November 2024.

Freshness

Last updated 2025-08-19 18:45:37.

Geography

null