Formal Language RNN-PPT: Synthetic Sequences for Language Model Pre-Pretraining

A 100,000-row subset of tokenized sequences for the EleutherAI/pythia-160m tokenizer, intended for language-model pre-pretraining experiments. The dataset was created by guox18 and last updated on May 20, 2026. Code and experiment scripts are available on GitHub.

Use Cases

• Pre-pretraining language models on formal-language sequences based on the dataset's stated purpose.
• Conducting curriculum learning experiments based on the mention of covering common RNN-PPT runs.
• Analyzing tokenization effects using the pre-tokenized sequences for a specific model tokenizer.

Strengths

• Contains a compact 100,000-row subset intended for easier download.
• Sequences are already tokenized for the EleutherAI/pythia-160m tokenizer.
• Associated code and experiment scripts are available on GitHub.

Limitations

• Row count, column definitions, and file formats are unknown, limiting suitability assessment.
• Description metadata is limited; actual data quality requires manual inspection after download.
• Column-level documentation is absent; field semantics must be inferred after download.

Provenance

Source

huggingface

Collection Method

RNN-generated for formal-language pre-pretraining.

Freshness

Last updated 2026-05-20 11:54:40.