LLM Versus Human Coding For Focus Group Analysis

93.5% mean agreement was achieved by large language models during deductive coding against an expert consensus adjudication. This dataset contains results from a blinded mixed-methods comparison of two LLMs (ChatGPT-5 and Claude 4 Sonnet), an LLM-based coding application, and human analysts on a focus-group transcript. It was authored by Callum Hill and published on figshare in April 2026.

Use Cases

• Compare deductive coding agreement metrics (e.g., 93.5% mean agreement, κ=0.34) between LLMs and human analysts.
• Evaluate inductive analysis performance using structured Likert-scale comparisons to a reference-standard set of expert-generated themes.
• Assess LLM hallucination rates (1.2% strict rate) and error rates from manual quote verification of model-coded segments.
• Benchmark model performance using Gwet's AC1 scores (e.g., 0.93 for LLM-assisted deductive analysis) against human coder scores.

Strengths

• Provides quantitative performance metrics, including agreement percentages with confidence intervals (e.g., 93.5%, 95% CI 92.5–94.5).
• Includes comparative results for multiple LLMs and a dedicated LLM-based application (QualiGPT) against blinded human analysts.
• Reports on multiple evaluation dimensions: deductive coding, inductive coding, and hallucination/error rates.

Limitations

• Analysis is based on a single focus-group transcript, limiting generalizability.
• Dataset size is 21.3 KB, indicating a very small scope confined to one study's results.
• Performance metrics are specific to a 10-code, 6-theme codebook and one digital health topic.

Provenance

Source

figshare, authored by Callum Hill.

Collection Method

Blinded mixed-methods comparison of LLMs and human analysts on an in-person focus-group transcript for an AI-enabled digital health proposal.

Freshness

Last updated April 2026.