NovelPrompts: English Safety Prompts for LLM Judge Evaluation

NovelPrompts is an English safety dataset built to test LLM judges' behavior when evaluating prompt safety. It contains 194 prompts and completions designed to require understanding of novel concepts appearing after July 2025. The dataset was created by anissa218 and last updated on Hugging Face in June 2026.

Use Cases

• Benchmarking LLM safety evaluators based on their ability to assess prompts containing novel concepts.
• Testing LLM robustness to new slang or word meanings based on the dataset's design.
• Evaluating LLM knowledge cutoff and temporal reasoning based on the post-July 2025 novelty requirement.
• Analyzing LLM judge behavior on ambiguous or context-dependent safety scenarios.

Strengths

• Contains 194 specific prompt-completion pairs designed for a targeted evaluation task.
• Prompts are constructed around a defined novelty criterion: concepts appearing after July 2025.

Limitations

• Row count is unknown, which may limit suitability assessment.
• Column-level documentation is absent; field semantics must be inferred after download.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

huggingface

Time Range

Concepts are novel if they appeared after July 2025.

Freshness

Last updated 2026-06-09 08:52:11; freshness should be verified.