Benchmarking LLMs on Publication Ethics Case Studies

Twelve authorship dispute cases from the Committee on Publication Ethics forum were used to evaluate two large language models. Kannan Sridharan published this comparative analysis in April 2026. The dataset contains scored model responses across seven evaluation domains.

Use Cases

• Compare model scores for 'Actionability of Recommendations' and 'Safety and Avoidance of Hallucination' between Gemini and DeepSeek.
• Analyze performance differences in the 'Identification of Ethical Issues' domain across three prompting strategies: Minimal, Deterministic, and Stochastic.
• Evaluate the 'Consistency with COPE Principles' scores to assess alignment with expert forum responses.
• Examine qualitative disagreement rates (8% major for Gemini, 16% combined for DeepSeek) with COPE advice.

Strengths

• Responses scored across seven distinct domains using a 5-point Likert scale.
• Analysis includes 12 real-world authorship and contributorship cases from COPE.
• Comparative evaluation of two specific LLM versions: Google Gemini 2.5 Flash and DeepSeek-V3.2.

Limitations

• Limited scope with only 12 cases analyzed.
• Model performance data is derived from a single study and may not generalize.
• The underlying case data is presented as a summary within a 120.9 KB PDF, not as structured data.

Provenance

Source

Committee on Publication Ethics (COPE) forum cases.

Collection Method

Cross-sectional analysis of LLM responses to case prompts, scored by independent raters.

Time Range

null

Freshness

Last updated April 2026.

Geography

null