Metaeval: NLP Task Collection for Extreme Multitask Learning

Metaeval is a collection of natural language processing tasks designed for meta-learning and extreme multitask learning, released by author sileod in early 2023. It aggregates various NLP benchmarks to facilitate the study of task-embeddings and model generalization across diverse linguistic challenges.

Use Cases

• Developing meta-learning algorithms that utilize task-embeddings to adapt to new NLP domains
• Benchmarking extreme multitask learning models across a wide variety of text classification or generation tasks
• Evaluating zero-shot transfer capabilities by training on a subset of the aggregated tasks

Strengths

• Aggregates multiple distinct NLP tasks into a single benchmark
• Released under the permissive MIT license
• Specifically tagged for task-embedding and meta-learning research

Limitations

• Lack of detailed documentation regarding specific column schemas or row counts
• No sample data available for immediate preview
• Potential for task overlap or redundancy depending on the source benchmarks included

Provenance

Source

sileod (GitHub)

Collection Method

Aggregation of existing natural language processing benchmarks

Freshness

Last updated in January 2023; no ongoing update schedule is provided.