Common Crawl Statistics: Monthly Web Crawl Metrics and Character Encoding Distributions

Common Crawl statistics provide metrics about its Monthly Crawl Archives, including the number of pages, distribution of top-level domains, and crawl overlaps. The dataset includes a 'Charsets' file showing the percentage of HTML pages encoded with different character sets, as identified by Tika's AutoDetectReader. This dataset is maintained by Common Crawl and was last updated on 2026-05-24.

Use Cases

• Analyze the evolution of web page character encoding usage based on the 'Charsets' statistics.
• Study the distribution and popularity of top-level domains across monthly web crawls.
• Measure the overlap and consistency between successive Common Crawl archives.
• Assess the scale of web crawls by tracking the number of pages crawled per month.

Strengths

• Data is derived from Common Crawl, a major source of open web crawl data.
• Includes specific metrics like character set percentages for HTML pages.
• Covers monthly archives, allowing for temporal trend analysis.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

Common Crawl web crawls.

Collection Method

Aggregated metrics and analysis from monthly crawl archives.

Freshness

Last updated 2026-05-24 20:03:32; freshness should be verified.