TID-8: Benchmark for Learning from Data with Inherent Disagreement

A benchmark dataset proposed in a paper at Findings of EMNLP 2023, focused on the task of letting models learn from data that has inherent disagreement. The dataset was created by MichiganNLP and last updated on October 30, 2023. It provides two splits, including an annotation split where each annotator's data is divided into train and test sets.

Use Cases

• Training models to handle inherent label disagreement based on the dataset's core focus.
• Benchmarking model performance on data with annotator disagreement based on the provided splits.
• Studying the effects of learning directly from conflicting annotations as described in the paper.

Strengths

• Dataset is associated with a peer-reviewed paper presented at Findings of EMNLP 2023.
• Provides a specific annotation split designed for studying annotator-level disagreement.

Limitations

• Description metadata is limited; actual data quality requires manual inspection after download.
• Column-level documentation is absent; field semantics must be inferred after download.
• Row count, file formats, and license information are unknown.

Provenance

Source

MichiganNLP

Collection Method

Likely aggregated from existing annotation tasks, as it is described as a benchmark.

Time Range

Associated with research published in 2023.

Freshness

Last updated 2023-10-30 18:18:31; freshness should be verified.

Geography

null