Dolma V1.7 30B: Tokenized Text Corpus for Llama 3 Pre-training

A tokenized version of the 30-billion-token Dolma V1.7 corpus, formatted for Llama 3. The dataset is split into 10 GB chunks for easier handling and was uploaded by user 'emozilla' on May 20, 2024. It is designed to be loaded directly as a NumPy memory-mapped array of integer token IDs.

Use Cases

• Pre-training or continuing pre-training of Llama 3-based models based on the tokenized input IDs.
• Benchmarking tokenizer efficiency or model perplexity on a large, pre-processed corpus.
• Analyzing the distribution and frequency of tokens in a large-scale web-derived text collection.
• Conducting research on data curation and filtering effects using a processed version of the Dolma corpus.

Strengths

• Contains 30 billion tokens from the Dolma V1.7 corpus, a substantial volume for model training.
• Pre-tokenized for the Llama 3 tokenizer, reducing preprocessing overhead for users of that model family.
• Split into 10 GB file chunks, which may facilitate distributed loading or processing on systems with memory constraints.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment for specific batch sizes.
• Description metadata is limited; actual data quality and content require manual inspection after download.

Provenance

Source

NousResearch/dolma-v1_7-30B

Collection Method

Tokenized version of the original Dolma corpus.

Freshness

Last updated 2024-05-20 01:36:01; freshness should be verified.