DCLM Data 300M: GPT-2-Tokenized Sequences for Data-Constrained Language Model Training

Pre-tokenized .pt files containing packed GPT-2-tokenized sequences derived from the DCLM corpus. The dataset snapshots were curated by author zhiwei555 for the paper 'Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws'. They were last updated on June 8, 2026.

Use Cases

• Training language models using Masked-Input Regularization (MIR) based on the described methodology.
• Studying scaling laws for language model pretraining under data constraints.
• Benchmarking model performance on pre-tokenized sequences derived from the DCLM corpus.

Strengths

• Data is pre-tokenized into GPT-2-tokenized sequences, which likely reduces preprocessing overhead.
• Dataset is explicitly curated for studying data-constrained language model pretraining, providing a focused resource.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

Derived from the DCLM corpus.

Collection Method

Packed GPT-2-tokenized sequences, curated as snapshots for a research paper.

Freshness

Last updated 2026-06-08 18:46:18; freshness should be verified.