Essential-Web v1.0: 24 Trillion Token Web Corpus with 12-Category Annotations

Essential-Web v1.0 is a 24-trillion-token corpus of web documents. Every document is annotated with a twelve-category taxonomy covering topic, format, content complexity, and quality. The dataset is hosted on AWS Open Data and was created by EssentialAI.

Use Cases

• Training large language models on a massive, categorized web corpus.
• Analyzing web content distribution based on the provided twelve-category taxonomy.
• Filtering or balancing training data for specific topics, formats, or quality levels.
• Benchmarking text classification models on the annotated categories.

Strengths

• Massive scale of 24 trillion tokens of web text.
• Every document is annotated with a twelve-category taxonomy.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count and specific data quality metrics are unknown.
• Last update date is unknown; freshness unverified.

Provenance

Source

EssentialAI

Collection Method

Likely gathered and processed from web archives.