LLM vs Human Deductive Coding Results for Focus Group Transcript

93.5% mean agreement was achieved by large language models during deductive coding of a focus-group transcript against an expert consensus adjudication. Callum Hill published this dataset in 2026, containing detailed performance metrics comparing LLMs and human analysts on qualitative coding tasks. The data supports a methodological framework for metric-based comparisons in thematic analysis.

Use Cases

• Compare deductive coding agreement metrics (e.g., mean agreement, κ, Gwet's AC1) between LLMs and human analysts.
• Analyze inductive coding performance using structured Likert-scale comparisons to a reference-standard set of inductive themes.
• Calculate hallucination and error rates from manual quote verification of LLM-coded segments.
• Benchmark specific LLMs (ChatGPT-5, Claude 4 Sonnet, QualiGPT) against blinded human coders using a 10-code, 6-theme codebook.

Strengths

• Provides precise statistical metrics, including 95% confidence intervals for agreement scores.
• Includes a strict hallucination rate of 1.2% (SD=2.1%) for inductive analysis.
• Compares multiple models and methods (deductive, inductive, verification) within a single study.

Limitations

• Limited scope, based on a single in-person focus-group transcript.
• Low overall code prevalence (7.8%, SD=3.2%) may affect generalizability of agreement statistics.
• Dataset size is only 58.4 KB, indicating a small, focused study.

Provenance

Source

figshare

Collection Method

Blinded mixed-methods comparison of LLMs and human analysts on a focus-group transcript for an AI-enabled digital health proposal.

Freshness

Last updated in April 2026.