Cdx Cc News: Common Crawl News Dataset Indexes

CDXj indexes for the Common Crawl News Dataset, created by a user named brian-learns. The indexes are organized by year and month, with a last recorded update timestamp of 2026-07-18. This dataset provides a structured way to look up news article captures from the Common Crawl archive.

Use Cases

• Build news article retrieval systems based on the SURT-formatted URLs optimized for lookup.
• Analyze temporal trends in news coverage using the capture_time timestamps.
• Extract and process news text from Common Crawl WARC files using the provided index records.

Strengths

• Indexes are organized by year and month, suggesting structured temporal partitioning.
• The dataset page indicates a last updated date of 2026-07-18, suggesting recent maintenance.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count, file size, and specific data formats are unknown, which may limit suitability assessment.
• The description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

Common Crawl News Dataset

Collection Method

User-created CDXj indexes derived from Common Crawl data.

Freshness

Last updated 2026-07-18 17:53:58; freshness should be verified.