LongBench v2: Reasoning Benchmark for 8k+ Token Contexts

LongBench v2 is a benchmark for evaluating Large Language Models on reasoning tasks involving context lengths of 8,000 tokens or more. Released in December 2024 by THUDM researchers, it contains approximately 1,000 samples across tasks like table question answering and text classification.

Use Cases

• Evaluating LLM reasoning capabilities using the multiple-choice question labels
• Benchmarking long-context retrieval and synthesis via the Table Question Answering tasks
• Testing text classification accuracy on documents exceeding 8,000 tokens

Strengths

• Context lengths starting at 8,000 tokens
• Covers multiple task types including Table QA and Text Classification
• Apache 2.0 licensed for open use

Limitations

• Small sample size of under 1,000 records
• Limited to English language tasks
• High computational requirements for processing 8k+ token contexts

Provenance

Source

THUDM (Tsinghua University) and zai-org

Collection Method

Expert-curated reasoning tasks from real-world multitasks

Freshness

Last updated December 2024.