HH-RLHF: Helpful and Harmless Reinforcement Learning from Human Feedback

Anthropic's HH-RLHF dataset contains between 100,000 and 1,000,000 human preference comparisons focused on model helpfulness and harmlessness, released in 2022. These text-based records are designed to facilitate the training of reward models for Reinforcement Learning from Human Feedback (RLHF) rather than supervised fine-tuning.

Use Cases

• Training reward models by comparing 'chosen' vs 'rejected' response pairs to score model outputs
• Fine-tuning language models for safety using the specific harmlessness preference labels
• Benchmarking alignment algorithms against the data used in the 2204.05862 research paper

Strengths

• Contains over 100,000 human-annotated preference pairs
• Provides dual-axis labels for both helpfulness and harmlessness dimensions
• Distributed under the permissive MIT license

Limitations

• Explicitly not recommended for supervised fine-tuning of dialogue agents as it may lead to poor performance
• Subject to the subjective biases of human annotators regarding what constitutes 'helpful' or 'harmless' behavior

Provenance

Source

Anthropic (arxiv:2204.05862)

Collection Method

Human annotation of model-generated responses.

Freshness

Last updated May 2023; based on research published in April 2022.