PILOT-Bench: 10K-100K IRAC-Aligned U.S. Patent Trial Decisions

PILOT-Bench provides between 10,000 and 100,000 records derived from U.S. Patent Trial and Appeal Board (PTAB) decision documents. Created by Yehoon Jang and TeamLab, this benchmark evaluates the structured legal reasoning of Large Language Models using IRAC-aligned classification tasks.

Use Cases

• Benchmarking LLM performance on IRAC-aligned classification tasks within patent law
• Training models to classify legal components such as 'Issue' or 'Rule' in PTAB documents
• Evaluating the consistency of legal reasoning in automated patent invalidation analysis

Strengths

• 10,000 to 100,000 record scale
• IRAC-aligned classification labels
• Sourced from official U.S. PTAB documents

Limitations

• Geographic bias toward the U.S. legal system
• Domain-specific focus on patent invalidation trials

Provenance

Source

U.S. Patent Trial and Appeal Board (PTAB)

Collection Method

Extracted from official PTAB decision documents

Freshness

Last updated March 2026

Geography

United States