Dspy Security Bench Trainset: Synthetic Queries for Prompt Injection Robustness

dspy-security-bench trainset (v0.1) is a synthetic dataset containing 192 query and ground-truth pairs. It was created by author immu4989 and last updated on June 29, 2026. The dataset is grounded in the AgentDojo workspace suite's default environment, which includes a calendar, inbox, and files.

Use Cases

• Optimizing DSPy program prompts for security based on the 192 synthetic query/response pairs.
• Benchmarking the effect of prompt optimization on prompt-injection robustness in agentic LLM programs.
• Training or fine-tuning models to recognize or generate secure prompts within a simulated workspace environment.
• Studying the characteristics of synthetic, environment-grounded queries designed to test LLM security.

Strengths

• Contains 192 precisely defined query and ground-truth pairs.
• Data is synthetically generated and grounded in a consistent, defined workspace environment (calendar, inbox, files).

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Freshness should be verified as the last update was 2026-06-29 06:26:33.

Provenance

Source

immu4989 on Hugging Face.

Collection Method

Synthetically generated, environment-grounded queries.

Freshness

Last updated 2026-06-29 06:26:33.