Alejandro Hernández-Arango's dataset provides Cohen's kappa coefficients with 95% confidence intervals for evaluating agreement. It includes per-criterion scores for inter-human agreement (κ = 0.408), NLP-V1 vs. human (κ = 0.510), and NLP-V2 vs. human (κ = 0.455). The 5.2 KB XLSX file was last updated on 2026-04-30.
Use Cases
- Benchmarking NLP model performance based on per-criterion agreement scores.
- Analyzing the reliability of human annotations using inter-human kappa coefficients.
- Comparing different NLP model versions (V1 vs. V2) based on their agreement with human raters.
- Assessing statistical confidence in agreement metrics using the provided 95% confidence intervals.
Strengths
- Provides three specific Cohen's kappa coefficients (0.408, 0.510, 0.455) for direct comparison.
- Includes 95% confidence intervals for each agreement metric, offering statistical context.
- Released under a permissive CC-BY-4.0 license, allowing for broad reuse.
Limitations
- Row count is unknown, which may limit suitability assessment.
- Column-level documentation is absent; field semantics must be inferred after download.
- The dataset is very small (5.2 KB), indicating limited scope.
Provenance
- Source
- Alejandro Hernández-Arango via figshare
- Freshness
- Last updated 2026-04-30 17:49:44; freshness should be verified.