PhishNChips: 2,000 Email Stimuli for LLM Agent Security Benchmarking

2,000 email stimuli and 220,000 adjudicated model evaluations comprise the PhishNChips benchmark for evaluating how system prompt configurations influence the security behavior of LLM-based email agents. The canonical v5.2 release was authored by AreLit and last updated on April 8, 2026.

Use Cases

• Benchmarking LLM phishing detection performance based on system prompt variations
• Analyzing the relationship between prompt configuration and agent security behavior based on the described stimuli
• Training or fine-tuning email security models using the adjudicated evaluation data

Strengths

• Large-scale benchmark with 2,000 distinct email stimuli
• Includes 220,000 adjudicated model evaluations for analysis

Limitations

• Column-level documentation is absent; field semantics must be inferred after download
• Row count for the primary stimuli is known, but other dataset dimensions like file formats are unknown

Provenance

Source

AreLit on Hugging Face

Collection Method

Likely constructed for benchmarking purposes; specific collection method not detailed.

Time Range

null

Freshness

Last updated 2026-04-08 18:04:40

Geography

null