Tiny Textbooks: 100K to 1M Synthetic Educational Text Records

Between 100,000 and 1,000,000 synthetic textbook-style text records generated by nampdn-ai in 2024 for training small language models. The dataset was produced using the Nous-Hermes-Llama2-13b model over 500 hours of compute time. It utilizes source material filtered from falcon-refinedweb and minipile to provide diverse educational content.

Use Cases

• Pre-training small language models using the synthetic text content to improve reasoning capabilities
• Fine-tuning models for educational text generation based on the textbook-style structure
• Evaluating synthetic data influence on model perplexity compared to the original falcon-refinedweb source

Strengths

• 100,000 to 1,000,000 records
• Generated via 500 hours of compute on 4x3090 Ti GPUs
• Source material filtered from falcon-refinedweb and minipile

Limitations

• Entirely synthetic content may contain factual hallucinations from the generator model
• Knowledge is restricted to the training cutoff of the Nous-Hermes-Llama2-13b model

Provenance

Source

nampdn-ai via Hugging Face

Collection Method

synthetic generation using Nous-Hermes-Llama2-13b

Freshness

Last updated 2024-07-03.