Betterdataset 2M: Mixed Pretraining Corpus for Language Models

Betterdataset 2M is a mixed pretraining corpus assembled by GODELEV from four open-source datasets. It contains approximately 2 million rows and an estimated 1.59 billion tokens, intended for training language models. The dataset was last updated on Hugging Face on June 1, 2026.

Use Cases

• Pretrain general-purpose language models based on the mixture of quality-filtered web text and educational content.
• Fine-tune models for mathematical reasoning tasks based on the inclusion of the finemath dataset.
• Train or fine-tune code generation models based on the filtered Python code functions component.
• Benchmark model performance on sequential instructional text based on the cosmopedia-derived content.

Strengths

• Comprises 2 million rows of data.
• Aggregates content from four established sources with specified weightings (60% FineWeb-HQ, 20% cosmopedia, 10% finemath, 10% python-stack).
• Includes an estimated 1.59 billion tokens, providing substantial training material.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• The license for the composite dataset is unknown, which may restrict usage.
• Freshness should be verified as the last update date is in the future (2026-06-01).

Provenance

Source

Aggregated from epfml/FineWeb-HQ, HuggingFaceTB/cosmopedia, HuggingFaceTB/finemath, and bigcode/python-stack-v1-functions-filtered.

Collection Method

Mixed and weighted combination of existing datasets.

Freshness

Last updated 2026-06-01 15:37:18; freshness should be verified.