ST-CORE-TOKENS: High-Density Tokenized Dataset for Indian LLMs

ST-CORE-TOKENS is an ultra-refined, high-density tokenized dataset developed by SKT AI LABS. The dataset is intended for training Indian large language models and is described as containing distilled logic. It was last updated on the platform in April 2026.

Use Cases

• Pre-training or fine-tuning Indian large language models based on the described high-density tokenization.
• Researching cognitive logic and reasoning in AI systems based on the dataset's stated purpose.
• Developing advanced natural language processing applications for Indian languages or domains.

Strengths

• Described as an 'ultra-refined, high-density' dataset, suggesting a focus on quality over raw volume.
• Explicitly developed for 'Indian LLMs' and 'Cognitive Logic', indicating a targeted application.

Limitations

• Description metadata is limited; actual data quality, format, and content require manual inspection after download.
• Column-level documentation, sample data, and specific size metrics are absent, hindering suitability assessment.
• The description contains promotional language, which may obscure objective technical details.

Provenance

Source

SKT AI LABS (author SKT-NRS)

Collection Method

Likely created through a distillation or tokenization process, as described.

Time Range

null

Freshness

Last updated 2026-04-08 19:59:22; freshness should be verified.

Geography

India (implied by focus on 'Indian LLMs' and 'Sovereign AI for India')