Comparative LLM Performance on Publication Ethics Case Evaluations

12 authorship and contributorship dispute cases from the Committee on Publication Ethics (COPE) forum were used to evaluate two large language models. The study, authored by Kannan Sridharan and shared in 2026, scored model responses across seven domains, including Actionability of Recommendations and Consistency with COPE Principles. Both Gemini and DeepSeek models achieved perfect scores in one domain but showed weaknesses in identifying specific ethical issues.

Use Cases

• Benchmark LLM performance on ethical reasoning by analyzing scores across seven domains like 'Identification of Ethical Issues' and 'Safety and Avoidance of Hallucination'.
• Compare prompting strategies (Minimal, Deterministic, Stochastic) by examining their impact on model scores for specific domains such as 'Overall Appropriateness'.
• Identify model-specific 'legal blind spots' by reviewing qualitative findings where models missed legal or copyright nuances in case responses.
• Extract novel 'value-add' strategies, such as author disassociation statements, proposed by the LLMs but absent from original COPE forum advice.

Strengths

• Direct comparison of two prominent LLMs (Gemini 2.5 Flash and DeepSeek-V3.2) using a structured 5-point Likert scale.
• Evaluation based on 12 real-world COPE forum cases, providing concrete grounding in publication ethics disputes.
• Analysis includes quantitative scores across seven distinct domains and qualitative metrics like an 8% major disagreement rate for Gemini.

Limitations

• Very small sample size of only 12 cases limits statistical power and generalizability of findings.
• Models demonstrated a noted weakness in the 'Identification of Ethical Issues' domain, with average scores below 4.
• The 291.2 KB PDF contains the study's results, not the underlying raw evaluation data or case texts.

Provenance

Source

Study authored by Kannan Sridharan, shared on figshare.

Collection Method

Cross-sectional analysis scoring LLM responses to 12 COPE authorship/contributorship cases using three prompting strategies.

Freshness

Last updated in April 2026.