CausalT5K: A Benchmark for Diagnosing Causal Reasoning in LLMs

CausalT5K is a benchmark dataset for evaluating causal reasoning in large language models. The dataset is a deduplicated, evaluation-ready JSON export created by GloriaGeng, with files last updated in May 2026. It is designed to diagnose specific reasoning failures like skepticism, sycophancy, and rung collapse.

Use Cases

• Benchmarking LLM performance on association-level (L1) causal reasoning tasks based on the described Pearl levels.
• Evaluating intervention-level (L2) causal reasoning in models using the dedicated JSON files.
• Studying specific failure modes like the detection-correction gap in model reasoning.
• Training or fine-tuning models on structured causal reasoning problems.

Strengths

• Dataset is explicitly deduplicated and described as 'evaluation-ready'.
• Contains 743 unique cases for Association (L1) reasoning and 3,302 for Intervention (L2) reasoning.
• Structured to diagnose specific, named reasoning failures (skepticism, sycophancy, rung collapse).

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count for the full dataset is unknown, which may limit suitability assessment.
• Description metadata is limited; actual data quality and structure require manual inspection.

Provenance

Source

GitHub repository genglongling/CausalT5kBench and associated arXiv paper.

Collection Method

Likely constructed for research purposes to benchmark LLMs, but specific gathering method is not detailed.

Freshness

Last updated 2026-05-22 02:28:43; freshness should be verified.