Toxic Conversation Annotations from Civil Comments Platform

Featuring 50,000 training examples of online comments from the Civil Comments platform, annotated for toxicity. Each comment was labeled by 10 annotators, with a comment considered toxic if the aggregated target score is >= 0.5. The dataset is imbalanced, with approximately 8% of comments marked as toxic.

Use Cases

• Train a text classifier to predict toxicity labels from comment text using the 50,000 annotated examples.
• Analyze inter-annotator agreement and label distribution for comments where the aggregated target score is >= 0.5.
• Study the characteristics of the 8% minority class of toxic comments to understand common linguistic patterns.

Strengths

• Contains 50,000 training examples of annotated text data.
• Each example was annotated by 10 different annotators, providing a measure of label reliability.
• The dataset is derived from the Jigsaw Unintended Bias in Toxicity Classification dataset, a known benchmark.

Limitations

• The dataset is imbalanced, with only about 8% of comments labeled as toxic, which can bias classification models.
• It contains only the first 50k examples from a larger training set, which may not be a representative sample.
• Specific column names and data structure are not provided in the input.

Provenance

Source

Jigsaw Unintended Bias in Toxicity Classification dataset, via the Civil Comments platform.

Collection Method

Comments were annotated by 10 annotators per example; toxicity was set when the aggregated target score >= 0.5.

Freshness

Last updated on 2022-05-13.

Geography

Region tag suggests 'us', but specific coverage is not confirmed.