LLM Performance Evaluation on Focus Group Coding

Callum Hill provides a mixed-methods comparison of large language models and human analysts on qualitative coding. The dataset contains results from evaluating two general-purpose LLMs and one specialized application on a focus-group transcript for an AI-enabled digital health proposal. The study was published on April 3, 2026.

Use Cases

• Compare deductive coding agreement metrics (e.g., 93.5% mean agreement, κ=0.34) between LLMs and human analysts using a 10-code, 6-theme codebook.
• Benchmark inductive analysis performance of LLMs against a reference-standard set of inductive themes generated by expert consensus.
• Calculate LLM hallucination rates (1.2% strict rate) and error rates by analyzing manual quote verification data on model-generated segments.
• Assess model non-inferiority in inductive analysis using reported p-values (p=0.043) from the structured Likert-scale comparison.
• Evaluate inter-rater reliability using Gwet's AC1 statistics (0.92 for humans, 0.93 for LLMs) for deductive coding on focus-group data.

Strengths

• Provides detailed quantitative metrics for model evaluation, including agreement percentages with confidence intervals (e.g., 93.5%, 95% CI 92.5–94.5).
• Employs a blinded, mixed-methods design comparing multiple LLMs and human analysts against an expert consensus adjudication.
• Reports specific hallucination (1.2%) and error rates derived from manual quote verification of LLM output segments.

Limitations

• The dataset is very small at 108.2 KB, indicating limited scope, likely containing only analysis results from a single focus-group transcript.
• Performance findings are based on one transcript from a specific domain (AI-enabled digital health), limiting generalizability.
• Low overall code prevalence (7.8%) in the deductive analysis may affect the interpretation of agreement statistics.

Provenance

Source

Callum Hill via figshare.

Collection Method

Blinded mixed-methods comparison of LLMs (ChatGPT-5, Claude 4 Sonnet, QualiGPT) and human analysts on a focus-group transcript.

Time Range

null

Freshness

Last updated on April 3, 2026, indicating recent analysis.

Geography

null