AI Ethics Preference Annotations for 95 Prompts and 190 Response Pairs

A human-annotated preference dataset for RLHF and Direct Preference Optimization (DPO), focused on AI ethics failure modes. It contains 95 prompts and 190 response pairs, with full annotation across five dimensions. The dataset was created by AI ethics specialist Mandy Hathaway and last updated on 2026-04-13.

Use Cases

• Training reward models for Reinforcement Learning from Human Feedback (RLHF) based on annotated preference pairs.
• Fine-tuning language models via Direct Preference Optimization (DPO) using the provided ethical preference data.
• Benchmarking model performance on AI ethics failure modes using the annotated prompt-response pairs.
• Analyzing ethical alignment in AI responses across the five annotation dimensions mentioned in the description.

Strengths

• Human-annotated by a specialist with an MA in Ethical Technology & Artificial Intelligence.
• Contains 95 prompts and 190 response pairs, providing a focused corpus.
• Annotations cover five distinct dimensions for detailed preference analysis.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Dataset size, file formats, and license information are not provided in the input.

Provenance

Source

huggingface

Collection Method

Human annotation by a specialist.

Time Range

null

Freshness

Last updated 2026-04-13 06:52:21; freshness should be verified.

Geography

null