SlimPajama 6B Token English Text Corpus

5,489,000 rows of English text data form a 6-billion-token subset of the larger SlimPajama-627B corpus. The dataset was created by DKYoon in August 2023, sampling 10% from a single chunk of the original shuffled data.

Use Cases

• Train a causal language model on 5.49 million rows of English text for next-token prediction.
• Fine-tune a text generation model using the 'en' language tag for English-specific tasks.
• Analyze text data proportions across source categories as defined in the original SlimPajama-627B documentation.
• Benchmark tokenizer efficiency and vocabulary coverage on a 6B-token corpus.

Strengths

• Contains 5,489,000 rows of text data.
• Represents a 6-billion-token sample, making it a substantial corpus for model training.
• Derived from the well-known SlimPajama-627B corpus, ensuring a foundation of diverse web text sources.

Limitations

• The dataset is a 10% sample of a single chunk from the original corpus, which may not be fully representative of the complete 627B-token distribution.
• No column or schema information is provided, complicating data loading and analysis.
• The validation and test sets are also sampled, potentially limiting their utility for standardized benchmarking.

Provenance

Source

Sampled from the cerebras/SlimPajama-627B corpus.

Collection Method

Created by downloading train/chunk1 of the original shuffled data and sampling 10% of its content.

Freshness

Last updated on 2023-08-21.

Geography

Region tag indicates 'us', suggesting a US focus, but specific spatial coverage is unknown.