Pile Document Toxicity Scores from Perspective API

A collection of a scored subset of 2.2 million documents from The Pile, processed through the Perspective API on May 18-20, 2022. It was created by tomekkorbak and provides toxicity annotations for text chunks.

Use Cases

• Analyze toxicity score distributions across 2.2 million text documents from The Pile.
• Train a text classifier to predict Perspective API toxicity scores for online content.
• Investigate correlations between document source metadata and assigned toxicity scores.

Strengths

• Derived from The Pile, a large-scale text corpus of 2.2 million documents.
• Toxicity scores are generated by the Perspective API, a widely-used tool for content moderation.
• Processing date is precisely documented as May 18-20, 2022.

Limitations

• The specific columns and data schema are unknown, complicating direct analysis.
• Toxicity scores are based on a single API snapshot and may not reflect current model versions.
• Lacks information on the original document text, requiring a join with The Pile source data.

Provenance

Source

The Pile, a large-scale text corpus.

Collection Method

Documents were scored using the Perspective API.

Time Range

Processing occurred on May 18-20, 2022.

Freshness

Data was last updated on 2022-05-20.

Geography

null