LLM Performance Evaluation on Publication Ethics Case Studies

A comparative evaluation scores the performance of Google Gemini 2.5 Flash and DeepSeek-V3.2 LLMs against expert Committee on Publication Ethics responses. The study analyzes 12 authorship and contributorship cases using three prompting strategies, with responses rated across seven domains on a 5-point Likert scale. The dataset was authored by Kannan Sridharan and published on figshare in April 2026.

Use Cases

• Compare LLM scores for 'Actionability of Recommendations' and 'Safety and Avoidance of Hallucination' domains across Gemini and DeepSeek models.
• Analyze performance differences in the 'Identification of Ethical Issues' domain under Minimal, Deterministic, and Stochastic prompting strategies.
• Examine qualitative disagreement rates, such as Gemini's 8% major disagreement with COPE versus DeepSeek's 16% combined disagreement rate.
• Assess mean similarity scores to COPE forum experts and identify unique 'value-add' strategies proposed by the LLMs.
• Evaluate model performance in the 'Consistency with COPE Principles' and 'Overall Appropriateness' domains for the 12 case studies.

Strengths

• Responses scored across seven distinct evaluation domains on a standardized 5-point Likert scale.
• Analysis includes 12 real-world COPE authorship and contributorship case studies.
• Performance comparison uses three distinct prompting strategies: Minimal, Deterministic, and Stochastic.

Limitations

• Limited scope with only 12 case studies analyzed, restricting generalizability.
• The dataset is a 121.8 KB PDF report, not a structured data file for direct computational analysis.
• Models demonstrated a 'legal blind spot,' missing specific legal and copyright nuances present in the cases.

Provenance

Source

figshare

Collection Method

Cross-sectional analysis of 12 COPE forum cases, with LLM responses scored by independent raters.

Freshness

Published in April 2026.