SKT Omni Corpus: A Large-Scale Multilingual Text Collection for LLM Training

A large-scale text corpus, likely containing trillions of tokens, built by SKT AI Labs as part of Project Surya. The dataset is intended to serve as a sovereign data foundation for indigenous large language model development in India. The dataset page indicates it is on a building stage with a target size of 8TB.

Use Cases

• Pre-training large language models based on the described trillions of tokens of text data.
• Developing or benchmarking multilingual NLP models based on the corpus's likely multilingual content.
• Researching data sovereignty and indigenous AI development based on the stated project goals.

Strengths

• Targets a very large scale of 8TB, suggesting substantial data volume.
• Explicitly designed for sovereign AI and indigenous LLM development, providing a specific use-case focus.

Limitations

• Description metadata is limited; actual data quality, content, and structure require manual inspection after download.
• Column-level documentation is absent; field semantics must be inferred after download.
• The dataset page states it is 'ON BULIDING STAGE', indicating it may be incomplete or unstable.

Provenance

Source

SKT AI Labs (SKT-NRS on Hugging Face).

Freshness

Last updated 2026-05-19 10:37:36; freshness should be verified.

Geography

Project focus is on India, but specific geographic coverage of the data is unknown.