Dolci Instruct DPO: 260,000 Preference Pairs for Olmo 3 Tuning

260,000 preference pairs for Direct Preference Optimization (DPO) developed by the Allen Institute for AI in 2025-2026. This mixture was utilized to preference tune the Olmo 3 Instruct 7B model using delta-aware heuristics and GPT-judge pipelines.

Use Cases

• Preference tuning of large language models using the 260,000 preference pairs
• Evaluating model alignment using the delta-aware preference heuristics
• Researching the impact of Delta Learning on instruction following

Strengths

• 260,000 preference pairs
• Used to train Olmo 3 Instruct 7B
• Includes delta-aware Ultrafeedback-esque GPT-judge annotations

Limitations

• Synthetic nature of GPT-judge annotations may introduce model-specific biases
• Heuristic-based pairs might lack the nuance of human-annotated preferences

Provenance

Source

Allen Institute for AI

Collection Method

Synthetic generation using Delta Learning heuristics and GPT-judge pipelines

Freshness

Last updated February 2026.