Tl 2B Pretok: Pretokenized Text Sequences for Language Model Training

Pretokenized chunks of text formatted as packed sequences of 513 tokens each, with no cross-document bleeding. The dataset was created by Beetle-Data and its metadata was last updated on May 18, 2026. It is sharded incrementally, with a marker file committed upon finalization.

Use Cases

• Training language models based on the provided 513-token input sequences.
• Benchmarking tokenization and sequence packing pipelines using the pretokenized format.
• Fine-tuning models on a corpus that has been preprocessed to avoid cross-document context bleeding.

Strengths

• Sequences are pretokenized into fixed-length chunks of 513 tokens.
• Explicitly designed to prevent cross-document bleeding in the packed sequences.
• Sharded and incrementally uploaded with a clear finalization marker.

Limitations

• Description metadata is limited; actual data quality requires manual inspection after download.
• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.

Provenance

Source

Beetle-Data

Freshness

Last updated 2026-05-18 12:38:09; freshness should be verified.