Autoresearch HITL Annotations: Human Evaluations of DPO Models

1,507 human annotations from a study titled 'When does autoresearch need a human?'. ProlificAI collected these evaluations from 300 participants assessing models generated by Karpathy's autoresearch on a DPO task. The dataset includes per-pair statistics, Bradley-Terry rankings, and LLM-clustered comment themes.

Use Cases

• Compare model performance based on Bradley-Terry rankings derived from human annotations.
• Analyze qualitative feedback themes from LLM-clustered participant comments.
• Study the effectiveness of automated research pipelines by evaluating the human annotations they require.

Strengths

• Dataset contains 1,507 individual annotation rows.
• Annotations were collected from 300 distinct human participants via Prolific.
• Includes multiple analysis outputs like Bradley-Terry rankings and clustered comment themes.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is known, but other scale details like file size and formats are unknown.

Provenance

Source

ProlificAI

Collection Method

Collected via the Prolific platform from 300 participants as part of a case study evaluating Karpathy's autoresearch.

Freshness

Last updated 2026-05-21 13:37:51; freshness should be verified.