Privacy Leakage Benchmark for Multi-Agent LLM Systems

AgentLeak is a full-stack benchmark for evaluating privacy leakage across the entire communication pipeline of multi-agent LLM systems. It was created by the author 'humain2' and the associated research paper and code were released around March 2026.

Use Cases

• Evaluating privacy leakage from intermediate agent communication traces using the benchmark's defined attack scenarios.
• Measuring the effectiveness of privacy defenses across different system components like prompts, agent memories, and final outputs.
• Benchmarking different LLM models against specific privacy attack vectors defined in the AgentLeak framework.
• Analyzing the correlation between data exposure in agent memory states and successful privacy inference attacks.

Strengths

• First full-stack benchmark for privacy in multi-agent LLM systems, covering the entire communication pipeline.
• Provides a systematic evaluation framework with defined attack scenarios and metrics.

Limitations

• Dataset size, row count, and specific column details are unknown.
• The benchmark's scope and generalizability may be limited to the specific multi-agent architectures and tasks it evaluates.

Provenance

Source

Research project 'AgentLeak' from the Privatris group.

Collection Method

Created as a benchmark for the associated academic paper.

Freshness

Last updated on the platform in March 2026.