WebGPT Human Preference Comparisons For Reward Modeling

OpenAI released a collection of human preference comparisons judged suitable for reward modeling, derived from the WebGPT project. The dataset was published in December 2022 and contains comparisons from the WebGPT paper.

Use Cases

• Train a reward model to rank text completions using the pairwise comparison labels.
• Fine-tune a language model using reinforcement learning based on the human preference data.
• Benchmark reward modeling algorithms on a curated set of comparisons from a large-scale language model project.
• Analyze human preferences for text quality, factuality, or helpfulness as captured in the WebGPT comparisons.

Strengths

• Data is curated specifically for reward modeling tasks.
• Source is from OpenAI's published WebGPT research, providing a known provenance.

Limitations

• Exact size (number of comparisons) is unknown, limiting assessment of statistical power.
• Potential bias towards the specific tasks and domains covered in the WebGPT experiments.
• Data is from 2022 and may not reflect more recent human preference distributions.

Provenance

Source

OpenAI, derived from the WebGPT research project.

Collection Method

Human comparisons collected during the WebGPT study, filtered for reward modeling suitability.

Freshness

Last updated on HuggingFace in December 2022.