SODA: Safety Over Depth for Agents Benchmark

SODA evaluates how conversation depth affects agent safety across 16 tool-use environments with 80 scenarios. Each task places a harmful request at a controlled depth from D=0 to D=20, preceded by regular agentic tasks. The dataset was created by author 'cesun' and was last updated on 2026-06-12.

Use Cases

• Benchmarking agent safety degradation based on conversation depth mentioned in the description
• Evaluating the effectiveness of safety guardrails in multi-turn tool-use environments described in the overview
• Studying the 'cold-start safety gap' for LLM agents as referenced in the dataset's purpose

Strengths

• Benchmark spans 16 distinct tool-use environments
• Contains 80 different scenarios for evaluation
• Systematically controls conversation depth from 0 to 20 steps

Limitations

• Column-level documentation is absent; field semantics must be inferred after download
• Row count is unknown, which may limit suitability assessment
• Description metadata is limited; actual data quality requires manual inspection after download

Provenance

Source

huggingface

Collection Method

Likely constructed for benchmarking purposes, as described.

Freshness

Last updated 2026-06-12 02:08:01; freshness should be verified