Dolma V1 7 3B: Tokenized Language Model Training Data for Llama 3

A tokenized version of the NousResearch/dolma-v1_7-3B dataset, formatted as a Nanotron dataset for efficient training. The dataset was uploaded by user 'emozilla' to Hugging Face on May 23, 2024. It is stored as a memory-mapped numpy array of token IDs, with the total number of tokens calculable from the file length.

Use Cases

• Train or fine-tune language models based on the tokenized Dolma corpus.
• Benchmark model training efficiency using pre-tokenized data in a memory-mapped format.
• Load and inspect token sequences directly using numpy for analysis or debugging.

Strengths

• Pre-tokenized for the Llama 3 tokenizer, which can save computational overhead during model training.
• Formatted as a memory-mapped numpy array, enabling efficient loading of large datasets.
• Derived from the Dolma corpus, a known open dataset for language model pre-training.

Limitations

• Description metadata is limited; actual data quality and content require manual inspection after download.
• Column-level documentation is absent; field semantics must be inferred after download.
• Row count and total dataset size are unknown, which may limit suitability assessment.

Provenance

Source

Derived from NousResearch/dolma-v1_7-3B, processed by user 'emozilla'.

Collection Method

Tokenized using the Llama 3 tokenizer and saved as a Nanotron-compatible numpy array.

Freshness

Last updated 2024-05-23 02:50:24; freshness should be verified.