Octobench

OctoBench is an instruction-following benchmark for coding agents, created by MiniMaxAI and released in January 2026. It is an extended version of OctoCodingBench, expanded from 72 manually annotated instances to 217 instances using AI-assisted augmentation. The dataset is hosted on Hugging Face and is intended for evaluating agentic coding performance.

Use Cases

• Benchmarking the instruction-following capabilities of coding agents based on scaffold-aware tasks.
• Evaluating agent performance on repository-grounded coding problems based on the dataset's augmented instances.
• Training or fine-tuning AI models for code generation based on the benchmark's structured tasks.

Strengths

• Contains 217 benchmark instances, expanded from an original 72 manually annotated examples.
• Specifically designed for scaffold-aware instruction following, a targeted evaluation niche.
• Released with an associated research paper on arXiv, providing methodological context.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment for large-scale training.
• Data may reflect bias inherent to the AI-assisted augmentation method described.

Provenance

Source

MiniMaxAI

Collection Method

AI-assisted augmentation of a manually annotated base dataset (OctoCodingBench).

Time Range

Dataset released in January 2026.

Freshness

Last updated 2026-01-16 07:40:26; freshness should be verified.

Geography

Regionus tag suggests a U.S. focus, but specific spatial coverage is unknown.