LLM Performance Scores on Publication Ethics Case Evaluations

12 authorship and contributorship cases from the Committee on Publication Ethics (COPE) forum were used to evaluate two large language models. The dataset contains scores across seven domains, including Actionability of Recommendations and Identification of Ethical Issues, rated on a 5-point Likert scale. Kannan Sridharan published this comparative analysis in April 2026.

Use Cases

• Compare model scores in the 'Identification of Ethical Issues' domain to assess LLM weaknesses in ethical reasoning.
• Analyze scores for 'Actionability of Recommendations' and 'Safety and Avoidance of Hallucination' to benchmark model reliability for editorial guidance.
• Examine performance differences under 'Minimal', 'Deterministic', and 'Stochastic' prompting strategies for prompt engineering research.
• Use the 8% and 16% disagreement rates with COPE experts to quantify alignment gaps between LLM and human ethical judgment.

Strengths

• Scores derived from 12 distinct COPE authorship and contributorship cases.
• Each case evaluated across seven specific domains, including Consistency with COPE Principles and Overall Appropriateness.
• Responses rated by independent raters using a standardized 5-point Likert scale.

Limitations

• Limited scope with only 12 cases, providing a narrow basis for generalization.
• Scores represent a single snapshot; model performance may vary with updates.
• The 95.0 KB PDF contains summarized results, not the raw, scored response data.

Provenance

Source

Cases sourced from the Committee on Publication Ethics (COPE) forum.

Collection Method

Cross-sectional analysis comparing LLM (Gemini 2.5 Flash, DeepSeek-V3.2) responses to expert COPE forum advice using three prompting strategies.

Freshness

Data reflects analysis current as of April 2026.