ArithMark 2.0: Procedurally Generated Integer Arithmetic Benchmark for Language Models

ArithMark 2.0 is a procedurally generated benchmark for evaluating integer arithmetic ability in language models. The benchmark is designed for base-model log-likelihood scoring and does not require instruction following or chain-of-thought reasoning. It was created by AxiomicLabs and was last updated on the Hugging Face platform in May 2026.

Use Cases

• Benchmarking language model performance on integer arithmetic based on the description of continuation-style multiple-choice problems.
• Evaluating base-model log-likelihood scoring for arithmetic expressions, as the benchmark is designed for this specific method.
• Studying procedural generation techniques for creating arithmetic evaluation datasets, as the data is procedurally generated.

Strengths

• Designed specifically for base-model log-likelihood scoring, a clear and defined evaluation protocol.
• Focuses on a specific capability (integer arithmetic) without confounding factors like instruction following.
• Procedurally generated, which likely allows for scalable and varied problem creation.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment for large-scale training.
• Description metadata is limited; actual data quality and problem diversity require manual inspection after download.

Provenance

Source

AxiomicLabs via Hugging Face.

Collection Method

Procedurally generated.

Freshness

Last updated 2026-05-22 03:57:13; freshness should be verified.