MegaMath Open Pretraining Corpus with Over 300 Billion Tokens

MegaMath is an open math pretraining dataset containing over 300 billion tokens, curated from diverse, math-focused sources. It was created by the LLM360 Team as part of TxT360, with data re-extracted from Common Crawl using math-oriented optimizations and filtering.

Use Cases

• Pretrain large language models on mathematical text using the over 300 billion tokens of curated content.
• Fine-tune models for mathematical reasoning tasks using the English-language, math-focused text sources.
• Apply fasttext-based filtering techniques similar to the dataset's curation process to other text corpora for quality improvement.
• Study the characteristics of mathematical documents extracted from web data via HTML optimizations and deduplication.

Strengths

• Contains over 300 billion tokens, providing a substantial scale for pretraining.
• Curated from diverse, math-focused sources to enhance domain relevance.
• Utilized math-oriented HTML optimizations and fasttext-based filtering during creation to improve data quality.
• Includes deduplication processes to reduce redundancy in the extracted documents.

Limitations

• Specific column structure, sample data, and file formats are not provided, limiting understanding of the data's internal organization.
• The dataset's size in gigabytes and exact row count are unknown, making storage and processing requirements unclear.
• Potential bias towards English-language mathematical content, as indicated by the 'English' tag, limiting multilingual applicability.

Provenance

Source

Diverse, math-focused sources, including documents re-extracted from Common Crawl.

Collection Method

Curated via math-oriented HTML optimizations, fasttext-based filtering, and deduplication of web data.

Freshness

Last updated on 2025-04-09.