LegalBench: 162 Legal Reasoning Tasks for LLM Evaluation

LegalBench contains 162 distinct tasks for evaluating legal reasoning in large language models, totaling between 10,000 and 100,000 records. Developed by Neel Guha and a community of legal experts, the collection was last updated in September 2024 and focuses on the United States legal system.

Use Cases

• Benchmarking model accuracy on legal text classification tasks across various legal domains
• Evaluating zero-shot performance on legal question answering using expert-curated prompts
• Testing legal text generation capabilities for tasks like contract clause interpretation

Strengths

• Contains 162 distinct sub-tasks for granular evaluation
• Scale of 10,000 to 100,000 records across multiple legal categories
• Expert-annotated tasks designed by legal professionals

Limitations

• Geographic bias toward the United States legal system
• Inconsistent task formats across the 162 different benchmarks

Provenance

Source

Neel Guha (nguha) and collaborators, associated with Arxiv 2308.11462

Collection Method

Collaborative expert curation and annotation

Freshness

Last updated September 30, 2024.

Geography

United States