Indic HPLT v2: Multilingual Pretraining Corpus Across 13 Indic Languages and English

34,605,630 documents across 13 Indic languages and English, built from the HPLT Monolingual v3 web crawl. This release is a larger successor to Indic HPLT v1, adding 3 new languages and approximately 3.5 times more documents. The dataset was uploaded by author AM0908 and last updated on 2026-05-19.

Use Cases

• Pretrain multilingual language models based on the corpus of 13 Indic languages and English.
• Benchmark cross-lingual transfer learning techniques based on the inclusion of multiple related languages.
• Analyze web-crawled text quality and characteristics for the HPLT Monolingual v3 source data.

Strengths

• Contains 34,605,630 documents, a significant scale for pretraining.
• Covers 13 Indic languages plus English, offering broad linguistic coverage.
• Represents an expansion from a previous version, adding Nepali, Odia, and Assamese.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Last updated 2026-05-19 23:30:42; freshness should be verified.

Provenance

Source

HPLT Monolingual v3 high-quality web crawl data.

Collection Method

Built from a web crawl.

Freshness

Last updated 2026-05-19 23:30:42.

Geography

Likely focused on regions where the 13 Indic languages are spoken.