ZEST: Zero-Shot Learning Benchmark with 25K Instances Across 1,251 Tasks

ZEST is a benchmark for zero-shot generalization to unseen NLP tasks, created by the Allen Institute for AI. It contains 25,000 labeled instances distributed across 1,251 different tasks. The dataset is hosted on AWS Open Data and is licensed under CC-BY-4.0.

Use Cases

• Benchmarking zero-shot learning models based on the 1,251 distinct NLP tasks.
• Training models for task understanding and generalization based on task descriptions.
• Analyzing model performance on unseen tasks based on the labeled instance structure.

Strengths

• 25,000 labeled instances provide a substantial evaluation corpus.
• 1,251 different tasks offer a broad testbed for zero-shot generalization.
• Explicitly designed as a benchmark for zero-shot learning in NLP.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Last update date is unknown; freshness unverified.

Provenance

Source

Allen Institute for AI

Collection Method

Benchmark dataset creation; specific collection method is not detailed.

Time Range

Temporal coverage is not specified.

Freshness

Last updated date is unknown.

Geography

Spatial coverage is not specified.