LLM Performance Scores on Publication Ethics Case Evaluations

12 authorship and contributorship cases from the Committee on Publication Ethics forum were used to evaluate two large language models. Kannan Sridharan published comparative performance scores across seven domains, including Actionability of Recommendations and Consistency with COPE Principles, in April 2026. The dataset contains Likert scale scores and qualitative disagreement rates from this cross-sectional analysis.

Use Cases

• Compare 'Actionability of Recommendations' and 'Safety and Avoidance of Hallucination' scores between Gemini and DeepSeek models.
• Analyze performance variation across prompting strategies (Minimal, Deterministic, Stochastic) using domain scores like 'Identification of Ethical Issues'.
• Examine the 'Consistency with COPE Principles' domain scores to assess model alignment with expert ethical reasoning.
• Study qualitative metrics such as the 8% major disagreement rate for Gemini and the 16% combined disagreement rate for DeepSeek.
• Use mean similarity scores (approximately 4) to COPE forum experts for benchmarking model performance in ethical case analysis.

Strengths

• Scores are derived from a structured evaluation of 12 distinct ethical cases.
• Performance is quantified across seven specific domains using a 5-point Likert scale.
• Includes qualitative analysis metrics like disagreement rates with expert responses.

Limitations

• The dataset is tiny (126.2 KB), containing only evaluation results for 12 cases, limiting statistical power.
• Model performance is based on a single, specific set of COPE forum cases, which may not generalize to other ethical dilemmas.
• The underlying case data (the 12 COPE cases) is not included, restricting analysis to the provided scores and metrics.

Provenance

Source

Analysis conducted by author Kannan Sridharan, using cases from the Committee on Publication Ethics (COPE) forum.

Collection Method

Cross-sectional analysis involving scoring model responses by independent raters.

Time Range

null

Freshness

Last updated April 2026.

Geography

null