News Article Summaries with Human Similarity Annotations

Part of the Massive Text Embedding Benchmark (MTEB) for evaluating summarization models. It contains news article summaries annotated by humans for semantic similarity estimation. The dataset is monolingual in English and originates from the US region.

Use Cases

• Evaluate embedding models on the semantic similarity of news article summaries using human-annotated scores.
• Benchmark summarization models against the MTEB framework for the 't2t' task category.
• Analyze the correlation between automated similarity metrics and human judgments on news summaries.

Strengths

• Dataset is part of the standardized Massive Text Embedding Benchmark (MTEB).
• Annotations are created by human annotators, providing a ground truth for similarity.
• The dataset page indicates a bug in the original evaluation script has been fixed in this version.

Limitations

• The specific number of rows, columns, and sample data are unavailable.
• The dataset's size category is listed as 'n1 K', suggesting it may contain only a few thousand data points.
• Geographic coverage is limited to the US region, which may not generalize to other locales.

Provenance

Source

https://github.com/Yale-LILY/SummEval

Collection Method

Human-annotated news article summaries.

Freshness

The dataset was last updated on 2025-05-03.

Geography

US