MBZUAI-LLM provides a split version of the SlimPajama-627B dataset, dividing its content based on original sources. The dataset consists of 15,967 JSONL files totaling approximately 883 gigabytes in compressed size. It was last updated on September 20, 2023.
Use Cases
- Study the effects of different data sources on model performance based on the split by 'redpajama_setname'.
- Combine subsets from specific sources for custom training tasks based on the pre-separated directories.
- Benchmark language models on text from distinct domains or languages likely contained in the corpus.
- Analyze the distribution and quality of text from different web crawls or repositories within the larger SlimPajama corpus.
Strengths
- The dataset is split into 15,967 individual files, which may facilitate parallel processing.
- Data is organized by source into separate directories, simplifying selective combination.
- The compressed size is approximately 883 gigabytes, indicating a substantial volume of text data.
Limitations
- Column-level documentation is absent; field semantics must be inferred after download.
- Row count is unknown, which may limit suitability assessment.
- Last updated 2023-09-20 06:26:19; freshness should be verified.
Provenance
- Source
- MBZUAI-LLM
- Collection Method
- Split of the cerebras/SlimPajama-627B dataset based on the 'redpajama_setname' field.
- Freshness
- 2023-09-20 06:26:19