TAB-Bench: Terminal Agent Task Alignment Benchmark

A benchmark suite of 89 terminal tasks derived from Terminal-Bench 2.1, created by author symbolorate and last updated on 2026-05-12. It measures whether a terminal agent correctly follows user instructions by using embedded cues while ignoring plausible distractors. Each task is intentionally underspecified to test selective information use.

Use Cases

• Benchmarking terminal agent performance based on the suite of 89 tasks.
• Evaluating an agent's ability to use helpful environmental cues based on the described task structure.
• Testing an agent's resistance to irrelevant distractors based on the described distractor mechanism.
• Researching task underspecification and instruction alignment in AI agents.

Strengths

• Contains 89 distinct terminal tasks for evaluation.
• Tasks are derived from the established Terminal-Bench 2.1 source.
• Each task includes a designed helpful cue and an irrelevant distractor.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

huggingface, author symbolorate

Collection Method

Derived from Terminal-Bench 2.1.

Freshness

Last updated 2026-05-12 14:19:42; freshness should be verified.