Benchmark Scores for LLM Evaluation of Publication Ethics Cases

A PDF document presents a cross-sectional analysis comparing the performance of Google Gemini and DeepSeek LLMs against expert COPE forum responses. The study includes 12 authorship and contributorship cases, with responses scored across seven domains on a 5-point Likert scale by independent raters. Author Kannan Sridharan published this 89.1 KB file on figshare in April 2026.

Use Cases

• Benchmarking LLM performance in ethical reasoning based on the seven scored domains (e.g., Actionability, Consistency with COPE Principles).
• Analyzing the impact of prompting strategies (Minimal, Deterministic, Stochastic) on LLM output quality for ethical case analysis.
• Comparing model disagreement rates with expert advice based on the reported 8% and 16% rates for Gemini and DeepSeek.
• Investigating model-specific strengths and weaknesses, such as the 'legal blind spot' and unique 'value-add' strategies mentioned in the results.

Strengths

• Scores are provided on a standardized 5-point Likert scale across seven specific domains.
• Analysis is based on 12 real-world authorship and contributorship cases from the COPE forum.
• Results include specific performance metrics, such as perfect scores (5 ± 0) in Actionability and high marks in Safety (4.88 ± 0.33).
• Qualitative disagreement rates with COPE experts are quantified (8% major for Gemini, 16% combined for DeepSeek).

Limitations

• The dataset is a 89.1 KB PDF; the underlying raw evaluation data or column-level details are not provided.
• Row count and data structure are unknown, which may limit suitability assessment for computational analysis.
• The study's scope is limited to 12 specific COPE cases and two LLM models.

Provenance

Source

Author Kannan Sridharan, published on figshare.

Collection Method

Cross-sectional analysis using three prompting strategies on 12 COPE cases, with responses scored by independent raters.

Time Range

The analysis covers cases from the COPE forum; specific temporal coverage of the cases is not stated.

Freshness

Last updated 2026-04-08 05:29:23

Geography

Not applicable; the data concerns publication ethics principles, not geographic data.