LLM Performance Scores on Publication Ethics Case Evaluations

12 authorship and contributorship dispute cases from the Committee on Publication Ethics forum were used to evaluate two large language models. Kannan Sridharan published this comparative analysis in April 2026. The dataset contains performance scores across seven evaluation domains for Google Gemini 2.5 Flash and DeepSeek-V3.2.

Use Cases

• Compare model scores in the 'Identification of Ethical Issues' domain to assess weaknesses in AI ethical reasoning.
• Analyze performance differences between 'Minimal', 'Deterministic', and 'Stochastic' prompting strategies on metrics like 'Actionability of Recommendations'.
• Examine the 'Consistency with COPE Principles' scores to gauge alignment between AI and expert human judgment.
• Use the 'Overall Appropriateness' and 'Safety and Avoidance of Hallucination' scores to benchmark model reliability for editorial support.
• Investigate the qualitative 'major disagreement rate' (8% for Gemini, 16% for DeepSeek) to understand where AI advice diverges from COPE forum experts.

Strengths

• Scores derived from 12 real-world COPE case studies, providing concrete evaluation scenarios.
• Performance measured across seven distinct domains (e.g., Actionability, Safety) using a standardized 5-point Likert scale.
• Includes qualitative analysis metrics such as a documented 8% major disagreement rate for one model.

Limitations

• Limited scope with only 12 cases, restricting statistical power and generalizability.
• Data is derived from a single evaluation study, not a longitudinal or multi-source collection.
• The 84.6 KB size indicates a small, summary-level dataset without raw model outputs or full case text.

Provenance

Source

figshare, authored by Kannan Sridharan.

Collection Method

Cross-sectional analysis scoring LLM responses to COPE case prompts against expert forum responses.

Time Range

Study publication date is 2026; specific case collection period is null.

Freshness

Last updated April 2026.