Human Preference Annotations for Image-to-Video Generation

DatapointAI released a 1,000-row dataset in March 2026 for evaluating image-to-video generation models. Each row contains a reference image, two generated videos from Pika and CogVideoX models, and 10 aggregated human preference annotations. The dataset provides a total of 10,000 individual human judgments on video quality.

Use Cases

• Training reward models for RLHF by learning from the aggregated human preference annotations between the Pika and CogVideoX video outputs.
• Benchmarking new I2V models by comparing their generated videos against the reference Pika and CogVideoX outputs using the provided human preference scores.
• Analyzing the correlation between specific visual features in the reference image and the human preference for one generated video over the other.
• Studying inter-annotator agreement and label quality by examining the 10 raw annotations aggregated per row into a majority vote.

Strengths

• Contains 1,000 distinct comparison instances for model evaluation.
• Provides 10,000 total human annotations, offering a substantial signal for preference learning.
• Annotations are aggregated via majority vote from 10 individual ratings per row, reducing noise.

Limitations

• The dataset is a small subset (1,000 rows) with larger versions (2,000 and 3,000 rows) available, limiting statistical power for some analyses.
• Potential for geographic or cultural bias in human preferences, as the annotator demographics are not specified.
• Limited to comparisons between only two specific model outputs (Pika and CogVideoX), which may not represent the full spectrum of I2V generation quality.

Provenance

Source

DatapointAI via Hugging Face.

Collection Method

Human annotations collected and aggregated via majority vote on pairs of videos generated from reference images by Pika and CogVideoX models.

Freshness

Last updated March 13, 2026.

Geography

Region tag indicates 'us', but specific annotator geography is unknown.