Dataflow Demo: Filtered Web Crawl Text for Pretraining

A 2026-01-05 demo dataset from OpenDCAI showing a text processing pipeline. It contains raw and filtered web page data from Common Crawl, with the filtered output file being 2.54 MB. The purpose is to demonstrate a pipeline for cleaning and structuring text data for pretraining.

Use Cases

• Benchmarking text filtering algorithms based on the raw-to-processed data comparison.
• Developing web text cleaning pipelines based on the demonstrated Common Crawl processing.
• Studying data quality metrics for pretraining corpora based on the described filtering pipeline.

Strengths

• Demonstrates a clear before-and-after data state with raw (206 MB) and filtered (2.54 MB) files.
• Provides a concrete example of processing Common Crawl data, a common web corpus source.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

Common Crawl

Collection Method

Web crawling and subsequent filtering/structuring via the DataFlow project pipeline.

Freshness

Last updated 2026-01-05 10:01:09; freshness should be verified.