Psychiatric Diagnostic Benchmark For Large Language Models

MentalBench is a benchmark dataset designed to evaluate the psychiatric diagnostic capabilities of large language models, created by author hysong. It provides a framework grounded in real-world psychiatric knowledge to test LLM reliability in a sensitive healthcare domain. The dataset was last updated on the platform in April 2026.

Use Cases

• Benchmarking LLM performance on psychiatric diagnostic questions using the benchmark's evaluation framework.
• Analyzing model outputs for specific psychiatric conditions referenced in the benchmark's test cases.
• Evaluating potential biases in LLM-generated diagnostic suggestions based on the provided real-world knowledge grounding.

Strengths

• Evaluation framework is grounded in real-world psychiatric knowledge.
• Dataset was updated on the platform in April 2026.

Limitations

• Specific sample size, column details, and data volume are unknown.
• Geographic and demographic coverage of the underlying psychiatric knowledge is unspecified.

Provenance

Source

hysong on Hugging Face.

Collection Method

Constructed as an evaluation benchmark, likely from psychiatric knowledge sources.

Freshness

Last updated on the platform in April 2026.