Legal Reasoning Benchmark Subset With 20 Samples Per Task

Legalbench Small is a subsampled version of the LegalBench benchmark for evaluating legal reasoning in large language models. It contains 3,180 total samples across 159 tasks, with 20 samples per task, formatted as zero-shot prompts with ground-truth answers. The dataset was created by nguha and updated in March 2026.

Use Cases

• Evaluate LLM performance on 159 distinct legal reasoning tasks using the provided zero-shot prompts and ground-truth answers.
• Benchmark model accuracy across diverse areas of law represented by the 159 tasks in the subset.
• Conduct cost-efficient evaluation of large reasoning models with the 20 samples per task structure.
• Analyze model failure modes on specific legal tasks within the curated 3,180-sample collection.

Strengths

• Provides a representative subset of 159 tasks from the full 162-task LegalBench benchmark.
• Contains 3,180 total samples structured with 20 samples per task for consistent evaluation.
• Formatted as zero-shot prompts with ground-truth answers, ready for LLM evaluation.

Limitations

• Small sample size of only 20 instances per task may limit statistical significance for some evaluations.
• As a subsampled version, it does not contain the full breadth of the original benchmark's data.
• Limited to English language text and may not represent legal reasoning in other jurisdictions.

Provenance

Source

Hugging Face dataset by author nguha.

Collection Method

Subsampled from the LegalBench benchmark for cost-efficient evaluation.

Time Range

null

Freshness

Last updated March 2026.

Geography

null