LLM-Generated vs Teacher-Written OSCE Stations for Medical Students

A comparative pilot study evaluating the quality of Objective Structured Clinical Examination stations generated by GPT-4o versus those written by medical teachers. The dataset likely contains expert ratings from 7 assessors on a 5-point Likert scale for 10 total stations. It was authored by Piotr Szychowiak and last updated on June 10, 2026.

Use Cases

• Benchmarking LLM performance in medical education content generation based on expert ratings.
• Analyzing the quality of LLM-generated clinical scenarios based on provided reference knowledge.
• Studying the reliability of LLM-generated assessment grids compared to human-written ones.
• Informing best practices for teacher review of AI-generated educational materials.

Strengths

• Direct comparison of 5 LLM-generated stations with 5 teacher-written stations targeting similar objectives.
• Expert evaluation by 7 assessors using a standardized 5-point Likert scale.
• Clear quality threshold defined: a station is 'good enough' if most assessors agree or strongly agree.

Limitations

• Description metadata is limited; actual data quality requires manual inspection after download.
• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.

Provenance

Source

Jeehp Dataverse

Collection Method

Likely contains expert ratings from a blinded comparative study.

Freshness

Last updated 2026-06-10 20:32:02; freshness should be verified.