JudgmentBench: Expert-Annotated Legal Evaluation Dataset for AI Feedback

JudgmentBench contains 30 real-world legal tasks with model-generated outputs at three quality levels. The dataset includes rubric scores and pairwise judgments from practicing lawyers, as well as annotations from GPT-5.4 and GPT-5.4-mini autograders. It was created by judgmentbench and last updated on May 7, 2026.

Use Cases

• Studying feedback protocol efficacy based on expert-annotated rubric scores and pairwise judgments.
• Benchmarking AI model performance on legal tasks based on outputs at three constructed quality levels.
• Evaluating autograder systems like GPT-5.4-mini against human expert annotations for legal work product.

Strengths

• Includes 30 distinct real-world legal tasks, providing a substantive scope.
• Features annotations from multiple sources: practicing lawyers and two GPT-5.4 model variants.
• Model outputs are provided at three constructed quality levels for comparative analysis.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

judgmentbench

Collection Method

Likely contains expert-annotated model outputs and autograder annotations for legal study assignments.

Time Range

null

Freshness

Last updated 2026-05-07 09:00:47; freshness should be verified.

Geography

null