AgentHarm: Safety Benchmark for LLM Agents with <1,000 Scenarios

AgentHarm consists of fewer than 1,000 test cases designed to measure the harmfulness of LLM agents, developed by the UK AI Safety Institute and Gray Swan AI in 2024. It provides a structured framework for evaluating how autonomous agents respond to malicious instructions across diverse domains.

Use Cases

• Evaluating agent safety by measuring completion rates of harmful scenarios
• Benchmarking model alignment using the provided malicious prompts
• Testing safety filters within agentic frameworks to prevent harmful task execution

Strengths

• Authored by safety researchers from the UK AI Safety Institute and Gray Swan AI
• Specifically targets agentic behavior rather than static text completion
• Includes peer-reviewed methodology from Arxiv 2410.09024

Limitations

• Small scale with fewer than 1,000 total records
• Restricted license categorized as 'other'
• Focus is limited to harmfulness, excluding general performance metrics

Provenance

Source

UK AI Safety Institute and Gray Swan AI

Time Range

2024

Freshness

Last updated December 2024.