HealthBench: OpenAI Medical Evaluation Data for Model Benchmarking

HealthBench provides the evaluation data for OpenAI's medical benchmarking suite, updated as of August 2025. It is designed to work with the simple-evals reference implementation to measure model performance in health-related domains.

Use Cases

• Measuring LLM accuracy on medical reasoning prompts
• Benchmarking health-domain performance using the simple-evals framework
• Testing model alignment with health-related reference answers

Strengths

• Official OpenAI evaluation set
• MIT licensed for open use
• Standardized for use with simple-evals framework

Limitations

• Metadata regarding specific column names and row counts is not provided in the repository description
• Limited to evaluation tasks rather than fine-tuning

Provenance

Source

OpenAI

Freshness

Last updated August 2025

Geography

United States