Human Preference Labels for AI-Generated Video Motion Quality

57,866 pairwise human preference labels compare 4 frontier video generation models. Datapoint AI collected these annotations across 3 quality dimensions for 417 unique prompts covering 11 motion categories. The dataset was last updated in March 2026.

Use Cases

• Training reward models for RLHF by learning from pairwise preference labels on video outputs.
• Benchmarking video generation models on specific motion categories using human-judged quality dimensions.
• Analyzing annotator agreement and bias in preference data collected for AI-generated content.

Strengths

• 57,866 pairwise comparison labels provide a substantial basis for preference modeling.
• Expanded scope includes 417 unique prompts and 11 motion categories, up from 60 prompts and 6 categories in a prior version.
• Labels are collected from real human annotators via a dedicated platform.

Limitations

• Specific sample size per model, prompt, or motion category is not provided, making granular analysis difficult.
• The dataset focuses on 4 specific frontier models, which may become outdated as new models are released.
• Potential for annotator subjectivity and bias inherent in human preference data.

Provenance

Source

Datapoint AI

Collection Method

Pairwise human preference labels collected from real annotators via the Datapoint AI platform.

Freshness

Last updated March 2026.

Geography

Region tags suggest US involvement, but specific coverage is not defined.