Global News Articles from Common Crawl Subset

Featuring 708,241 English-language news articles extracted from the Common Crawl CC-NEWS corpus using the news-please tool. It represents a processed subset of a global web crawl, focusing on English text from various news sites.

Use Cases

• Train a text classification model on the extracted article text to categorize news by topic or sentiment.
• Analyze linguistic patterns and named entities within the English-language article text for NLP research.
• Use the article text as a large corpus for pre-training or fine-tuning language models.
• Perform temporal analysis on article publication dates, if available in the metadata, to study news trends.

Strengths

• Contains 708,241 individual news articles, providing a substantial text corpus.
• Data is sourced from the Common Crawl, a large-scale, global web archive.
• Focuses on a processed English-language subset, which is a common requirement for NLP tasks.

Limitations

• The dataset is a small, unspecified portion of the full CC-NEWS corpus, limiting representativeness.
• Specific column structure, metadata fields, and data quality (e.g., extraction errors, duplicates) are unknown.
• Data is from 2022 or earlier, so it does not contain recent news events.

Provenance

Source

Common Crawl CC-NEWS bucket on AWS S3.

Collection Method

Extracted using the news-please tool (Hamborg et al., 2017) to collect the English-language portion.

Freshness

Last updated in April 2022; temporal coverage of the articles is unspecified.

Geography

Global, from news sites worldwide.