Legal Scenarios: SCOTUS 2024 Decisions for LLM Reasoning Evaluation

This text dataset contains fewer than 1,000 legal reasoning scenarios derived from U.S. Supreme Court slip opinions for the 2024 term. Created by relai-ai, it presents realistic legal conflicts requiring resolution and justification based on contemporary judicial precedent.

Use Cases

• Benchmarking LLM reasoning by evaluating model-generated justifications against SCOTUS resolutions
• Fine-tuning models for legal question-answering using scenario-based text
• Testing model awareness of 2024 legal precedents through zero-shot evaluation

Strengths

• Sourced from official 2024 U.S. Supreme Court slip opinions
• Designed for complex reasoning tasks involving competing legal principles
• Openly licensed under CC BY 4.0

Limitations

• Small sample size of fewer than 1,000 records
• Strict geographic bias to the United States federal legal system
• Undocumented column schema in the primary metadata

Provenance

Source

U.S. Supreme Court official website (slip opinions)

Collection Method

Extracted from 2024 term slip opinions by relai-ai

Time Range

2024

Freshness

Last updated May 2025, covering the 2024 judicial term.

Geography

United States