Human Preference Data from Chatbot Arena Battles

Over 55,000 real-world user and LLM conversations with associated user preferences, collected from battles between over 70 state-of-the-art LLMs. It was created for a Kaggle competition to predict human preferences in chatbot responses.

Use Cases

• Predict user preference labels (prefer model A, prefer model B, tie) from conversation text and model identifiers.
• Analyze the performance and user preference patterns across the over 70 LLMs mentioned, such as GPT-4 and Claude 2.
• Train a model to score or rank LLM responses based on the outcomes of head-to-head battles.

Strengths

• Over 55,000 real-world conversation samples provide a substantial base for analysis.
• Includes preferences across a wide range of over 70 state-of-the-art LLMs.
• Captures direct human preference labels from comparative battles, a key signal for alignment.

Limitations

• The specific features (columns) of the dataset are not detailed in the provided input.
• Potential for bias based on the specific user population interacting with the Chatbot Arena platform.
• The temporal scope and update frequency of the data are not specified.

Provenance

Source

lmarena-ai via Hugging Face.

Collection Method

Collected from real-world user interactions and battles on the Chatbot Arena platform.

Freshness

Last updated on 2024-05-17.