Long-Data-Col-rp_pile_pretrain: Filtered Long Text Corpus for Pretraining

Long-Data-Col-rp_pile_pretrain is a subset of the togethercomputer/Long-Data-Collections dataset, specifically the pretrain split files rp_sub.jsonl.zst and pile_sub.jsonl.zst. The dataset was created by BEE-spoke-data and last updated on December 29, 2025. To focus on long text, rows where the text contains fewer than 250 characters were dropped.

Use Cases

• Pretraining language models based on long text sequences.
• Benchmarking model performance on long-context understanding tasks.
• Studying the distribution and characteristics of lengthy text samples from the source collections.

Strengths

• Curated for long text, with all samples containing more than 250 characters.
• Derived from a known source dataset, togethercomputer/Long-Data-Collections.
• Last updated on December 29, 2025.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

togethercomputer/Long-Data-Collections

Collection Method

Subset extraction and filtering of the pretrain split.

Freshness

Last updated 2025-12-29 04:37:35.