Pg19 And Proof Pile: Pre-tokenized Evaluation Splits for Long-Context LLMs

Pre-tokenized binary evaluation splits used in the HiCI paper for hierarchical construction-integration in long-context LLMs. The dataset, authored by ZengXiangyu and last updated on 2026-04-21, contains PG19 test and validation sets tokenized with both Llama-2 and Llama-3 tokenizers.

Use Cases

• Benchmarking long-context language model performance based on the PG19 book corpus mentioned in the description
• Evaluating tokenizer-specific effects on model evaluation based on the provided Llama-2 and Llama-3 tokenized versions
• Testing hierarchical construction-integration methods for LLMs based on the stated purpose from the HiCI paper

Strengths

• Provides pre-tokenized data for two major tokenizers (Llama-2 and Llama-3), reducing preprocessing overhead
• Contains dedicated test and validation splits for the PG19 corpus, enabling standard evaluation protocols
• Data is specifically designed for evaluating hierarchical construction-integration methods as referenced in the HiCI paper

Limitations

• Column-level documentation is absent; field semantics must be inferred after download
• Row count is unknown, which may limit suitability assessment
• Description metadata is limited; actual data quality requires manual inspection after download

Provenance

Source

HuggingFace dataset by ZengXiangyu

Collection Method

Pre-tokenized from the PG19 corpus

Time Range

null

Freshness

Last updated 2026-04-21 14:56:23; freshness should be verified

Geography

null