Patient Safety Benchmarks For Language Model Evaluation

PatientSafetyBench is a dataset of synthetic prompts for evaluating safety behaviors in language models, particularly in medical contexts. It was created by Microsoft and was last updated in January 2026. The prompts are designed to test model responses to potentially harmful or unscientific medical content.

Use Cases

• Evaluating model refusal behavior on harmful medical prompts containing offensive or discriminatory language.
• Testing model adherence to scientific consensus by analyzing responses to prompts with unscientific medical claims.
• Benchmarking safety guardrails by measuring how models handle synthetic patient safety scenarios.

Strengths

• Created by a major AI research organization (Microsoft).
• Specifically designed for the critical domain of patient safety evaluation.
• Dataset page indicates active maintenance with a 2026 update date.

Limitations

• Contains only synthetic prompts, not real patient data or model interaction logs.
• The specific number of prompts, rows, and data size are unknown.
• Potential for limited scope if prompts do not cover a wide range of safety edge cases.

Provenance

Source

Microsoft

Collection Method

Synthetic prompt generation for evaluation purposes.

Freshness

Last updated January 2026.