Sutra 10B: 10 Billion Tokens of Synthetic Pedagogical Pretraining Data

Sutra 10B contains 10,193,029 educational entries totaling over 10 billion tokens, released by codelion in March 2026. This synthetic pedagogical dataset is generated via the Sutra framework to provide structured, multi-domain content for pretraining small language models.

Use Cases

• Pretraining small language models using the 10 billion tokens of pedagogical text
• Fine-tuning models for educational text generation using the structured entry format
• Benchmarking the performance of synthetic data against human-authored corpora in LLM training

Strengths

• Contains 10,193,029 discrete educational entries
• Scale of 10 billion tokens supports full pretraining of small language models
• Apache 2.0 license permits commercial and research use
• Multi-domain coverage across various educational subjects

Limitations

• Synthetic nature may introduce artifacts or biases inherent to the Sutra generation framework
• Lack of human-verified ground truth for pedagogical accuracy
• Undocumented column structure requires initial exploratory data analysis

Provenance

Source

codelion

Collection Method

synthetic

Freshness

Last updated March 2026.