Multilingual News Articles From Common Crawl

A collection of about 600 million news articles in over 100 languages, sourced from Common Crawl's WARC files. The dataset was processed, cleaned, deduplicated, and had language detection applied by the Stanford OVAL group. It covers articles from the beginning of 2016 through June 2024.

Use Cases

• Training multilingual language models using the article text and detected language labels.
• Analyzing global news trends over time using the article publication dates spanning from 2016 to 2024.
• Studying cross-lingual information diffusion by comparing article content across the over 100 represented languages.
• Building news recommendation or summarization systems based on the cleaned article text.

Strengths

• Approximately 600 million articles provides a massive scale for training.
• Covers a temporal range of over 8 years, from 2016 to mid-2024.
• Multilingual scope includes content in more than 100 languages.

Limitations

• Specific data quality metrics, column definitions, and sample sizes are not provided.
• The global scope may introduce geographic and linguistic biases based on Common Crawl's coverage.
• Data freshness is limited to articles crawled up to June 2024.

Provenance

Source

Common Crawl's CCNews Corpus.

Collection Method

Processed from all WARC files, with cleaning, deduplication, and language detection applied.

Time Range

2016-01-01 to 2024-06-30

Freshness

Last dataset update was 2024-08-31; article content is current through June 2024.

Geography

Global