Uzbek Web And Telegram Crawl Corpus From 1.2 Million Sources

UzCrawl contains web and Telegram crawl materials from nearly 1.2 million unique sources in the Uzbek language. The dataset is updated to March 2024 and was created to support research on low-resource languages. It is a large-scale text corpus assembled by the author tahrirchi.

Use Cases

• Train language models on the Uzbek language using a large-scale text corpus from diverse web and Telegram sources.
• Analyze linguistic patterns and topic evolution within the Uzbek Telegram corpus up to March 2024.
• Benchmark machine translation or text generation models for low-resource languages using this specific crawl data.

Strengths

• Contains materials from nearly 1.2 million unique sources, providing significant scale.
• Specifically targets the low-resource Uzbek language, filling a notable gap.
• Updated to include data current to March 2024, improving temporal coverage.

Limitations

• The specific number of rows, columns, and file formats is unknown, hindering precise technical assessment.
• Potential for noise, bias, or inconsistent quality inherent in large-scale web and social media crawls.
• Lack of detailed documentation on source filtering, preprocessing, or data structure.

Provenance

Source

Web and Telegram crawls.

Collection Method

Crawling of nearly 1.2 million unique sources.

Time Range

Data is up to date to March 2024.

Freshness

Updated to March 2024; last metadata update was 2024-06-03.

Geography

Primarily Uzbek language content, suggesting focus on Uzbekistan and related regions.