Legal LLM Benchmark: Safety and Utility Evaluations for 12 Models

1,000 to 10,000 records benchmark safety-utility trade-offs across 12 Large Language Models in the legal domain, published by marvintong in 2025. The data includes legal questions, multi-phase evaluations, and contract text to measure model performance and over-refusal tendencies. It is structured into distinct subsets for questions, evaluations, and legal documents.

Use Cases

• Benchmarking legal question-answering accuracy using the 'questions' subset
• Measuring model over-refusal rates by analyzing 'phase1_evaluations' and 'phase3_evaluations'
• Testing legal document understanding and information extraction using the 'contracts' data

Strengths

• Evaluates 12 different Large Language Models
• Scale of 1,000 to 10,000 records
• Includes multi-phase evaluation data (Phase 1 and Phase 3)

Limitations

• Geographic scope is restricted to the United States legal system
• Lack of detailed column-level documentation in the primary summary

Provenance

Source

marvintong via Hugging Face

Freshness

Last updated in November 2025.

Geography

United States