CCNews: 600 Million Multilingual News Articles from 2016 to 2024

600 million news articles from the Common Crawl archive, processed from 2016 to June 2024. The data has been cleaned, deduplicated, and includes language detection for articles in over 100 languages. This dataset was created by kareenamehta and is hosted on Hugging Face.

Use Cases

• Train large language models on a vast corpus of multilingual news text.
• Conduct cross-lingual information retrieval based on the language-labeled articles.
• Analyze global news trends and topics over time from 2016 to 2024.
• Benchmark text deduplication and cleaning algorithms on web-scale data.

Strengths

• Contains approximately 600 million articles, providing massive scale for training.
• Covers a temporal range from 2016 to June 2024, offering nearly a decade of news.
• Includes articles in more than 100 languages, supporting multilingual analysis.
• Data has been cleaned and deduplicated, which may reduce noise.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Data may reflect geographic and temporal bias inherent to the source Common Crawl archive.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

Common Crawl (CCNews Corpus)

Collection Method

Processing of WARC files from Common Crawl, with cleaning, deduplication, and language detection.

Time Range

2016 to June 2024

Freshness

Last updated 2026-03-18 18:27:12; freshness should be verified.

Geography

Global