DAFT Math: 199 Challenging Free-Response Problems for LLM Evaluation

199 challenging mathematical problems designed to be at the limit of current LLM abilities. The dataset, named DAFT Math (Difficult Automatically-scorable Free-response Tasks for Math), was created by metr-evals and last updated on July 17, 2025. It is presented as a research artifact for a niche use-case.

Use Cases

• Benchmarking LLM performance on difficult mathematical reasoning based on the described challenging problems
• Developing automated scoring methods for free-response math tasks based on the dataset's design
• Studying the boundaries of LLM abilities in mathematics based on tasks chosen at their limit

Strengths

• 199 specifically curated problems designed to be challenging for LLMs
• Tasks are free-response and automatically-scorable, as described in the dataset name

Limitations

• The dataset is described as having important limitations and is not a formal benchmark
• Column-level documentation and sample data are unavailable, requiring inspection after download
• Row count and file formats are unknown, which may limit suitability assessment

Provenance

Source

metr-evals

Freshness

Last updated 2025-07-17 15:04:56