Agent Trajectory Safety Benchmark With Fine-Grained Annotations

ATBench is a trajectory-level dataset for evaluating agentic safety in realistic, long-horizon interactions. It contains 500 annotated execution trajectories split evenly between safe and unsafe examples, with an average of 8.97 multi-turn interactions and 1,575 unique tools. The benchmark was created by AI45Research and was last updated in January 2026.

Use Cases

• Classify trajectory safety labels using features like turn count and tool usage patterns.
• Analyze the relationship between specific tool calls within a trajectory and the resulting safety annotation.
• Benchmark the performance of safety classifiers on the provided 250 safe and 250 unsafe trajectory examples.
• Study patterns in multi-turn interactions, leveraging the average of 8.97 turns per trajectory.

Strengths

• Contains 500 annotated trajectories for a balanced initial evaluation.
• Provides fine-grained, taxonomy-grounded safety annotations for precise analysis.
• Includes 1,575 unique tools, offering diversity in agent action space.

Limitations

• Limited to 500 total trajectories, which may be insufficient for training large models.
• The geographic and application domain bias of the trajectories is unknown.
• Relies on annotations which may contain subjective judgment or label noise.

Provenance

Source

AI45Research via Hugging Face.

Collection Method

Annotated execution trajectories from agent interactions; specific collection method not detailed.

Freshness

Last updated 2026-01-27.