Symbolic Pretraining Pile: Procedurally Generated Data for Formal Reasoning

The Symbolic Pretraining Pile (SPT) is a dataset for symbolic and formal pre-training, mid-training, and supervised fine-tuning. It is procedurally generated on CPU and can be scaled to trillion tokens, with adjustable difficulty. The dataset was created by reasoning-core and last updated on March 23, 2026.

Use Cases

• Pre-training language models on formal reasoning tasks based on the described planning and conjecture entailment categories.
• Supervised fine-tuning for logic-based natural language inference using the logic_nli task category.
• Training models for mathematical computation based on the described arithmetic and equation system tasks.
• Generating synthetic data for proof reconstruction and evidence retrieval tasks as mentioned in the description.

Strengths

• Data is procedurally generated, allowing for scalable creation up to trillion tokens.
• Task difficulty is adjustable via a single parameter, enabling controlled experimentation.
• Covers multiple formal reasoning categories including planning, logic, and mathematical computation.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count and file size are unknown, which may limit suitability assessment.
• Data is synthetic and procedurally generated, which may not reflect real-world complexity or distribution.

Provenance

Source

huggingface

Collection Method

Procedurally generated on CPU.

Time Range

Synthetic data; no temporal coverage.

Freshness

Last updated 2026-03-23 19:19:21.

Geography

Synthetic data; no spatial coverage.