Dolma3 Mix 150B: A 150-Billion-Token Multisource Text Corpus

A 150-billion-token sample from the Dolma 3 dataset pool, created by AllenAI and last updated in January 2026. It is a mixture of text from web pages, academic documents, code repositories, and scientific papers. The full dataset pool includes sources like Common Crawl, arXiv, and Stack-Edu.

Use Cases

• Training language models on a diverse mix of text sources based on the described web, academic, and code content.
• Benchmarking model performance across different text domains like web text and scientific papers.
• Analyzing the linguistic characteristics of text from sources like OCR-processed PDFs and GitHub code.
• Pre-training models for tasks requiring knowledge from mixed sources, including mathematics and Wikipedia.

Strengths

• Contains 150 billion tokens, providing substantial volume for model training.
• Sources are specified, including 121B tokens from Common Crawl and 19.9B from academic PDFs.
• Represents a mix of domains, including web text, academic papers, and code.

Limitations

• Description metadata is limited; actual data quality requires manual inspection after download.
• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.

Provenance

Source

AllenAI

Collection Method

Aggregated from multiple sources including Common Crawl, olmOCR, Stack-Edu, arXiv, and FineMath.

Freshness

Last updated 2026-01-15 05:24:49; freshness should be verified.