Urdu News Sentences from Leipzig Wortschatz 2016 Crawl

A collection of Urdu language sentences from a 2016 news crawl conducted by Leipzig Wortschatz. It is a subset of the ROOTS indic-ur corpus, representing 0.0587% of the total ROOTS data and 20.7635% of the indic-ur subset. The data has undergone deduplication and template removal processing.

Use Cases

• Train a language model on Urdu news text to generate or complete sentences.
• Analyze Urdu sentence structure and syntax patterns from news sources.
• Study the vocabulary and word frequency in modern Urdu news writing.

Strengths

• Data is sourced from the established Leipzig Wortschatz project.
• Processing steps include dedup_document and dedup_template_soft to reduce redundancy.
• The dataset is part of the larger, structured ROOTS corpus for Indic languages.

Limitations

• The dataset size is described only as a percentage of larger corpora, with no absolute row count provided.
• The sample data and specific column structure are unavailable for inspection.
• The data is from 2016, which may not reflect contemporary Urdu language usage.

Provenance

Source

Leipzig Wortschatz Crawl, processed by BigScience.

Collection Method

Web crawl of news sources, followed by deduplication and filtering.

Time Range

2016

Freshness

The dataset was last updated on the platform in December 2022, but the underlying news crawl is from 2016.

Geography

Region tag indicates 'us', but the language is Urdu; specific geographic coverage is unclear.