Annotated AI Benchmark Items with Cognitive Demand Scores

ADeLe Battery V1.0 contains 16,108 test items annotated with demand levels across 18 cognitive ability dimensions. The dataset was created by CFI-Kinds-of-Intelligence using GPT-4o annotators to apply the DeLeAn rubrics to existing AI benchmarks and was last updated in June 2025.

Use Cases

• Evaluate model performance on specific cognitive dimensions like 'logical reasoning' or 'attention and scan'.
• Benchmark AI systems against human-like ability levels (0-5+) across 18 distinct dimensions.
• Analyze the 'unguessability' dimension to assess the predictability of benchmark items.
• Study the distribution of cognitive demands across 63 source tasks.
• Train models to predict an item's demand level from its textual content.

Strengths

• 16,108 annotated items provides a substantial test set
• Labels span 18 distinct cognitive ability dimensions plus an unguessability score
• Items sourced from 63 diverse benchmark tasks

Limitations

• Specific column names and data structure are not provided
• Unknown if the dataset includes the original item text or only annotations
• Potential bias from the GPT-4o annotation methodology

Provenance

Source

CFI-Kinds-of-Intelligence

Collection Method

Created by applying DeLeAn rubrics via GPT-4o annotators to existing AI benchmarks.

Freshness

Last updated June 2025