Natural Language Inference Benchmark for 36 Linguistic Phenomena

Curriculum is a benchmark for evaluating language models on broad-coverage linguistic phenomena. It contains a collection of datasets covering 36 major types of linguistic phenomena and provides a diagnostic evaluation procedure.

Use Cases

• Diagnose model behavior on specific linguistic phenomena types defined in the benchmark.
• Verify language model learning quality using the evaluation procedure for distinct reasoning skills.
• Evaluate model performance across the 36 major linguistic phenomena categories.

Strengths

• Covers 36 major types of linguistic phenomena for broad-coverage evaluation.
• Provides a structured evaluation procedure for diagnosing model reasoning skills.

Limitations

• The specific datasets, their sizes, and internal structure are not detailed.
• The benchmark's composition and sample data are unavailable for inspection.

Provenance

Source

huggingface

Collection Method

Introduced as a new format of NLI benchmark, collection of datasets.

Freshness

Last updated in July 2022.