CALVIN Benchmark for Language-Guided Robot Manipulation

CALVIN is a benchmark dataset for language-conditioned policy learning in long-horizon robot manipulation tasks. The dataset was created by Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard, and the associated research was published in IEEE Robotics and Automation Letters in 2022. It is structured into tasks containing video and JSONL data.

Use Cases

• Train vision-language-action models using paired video and language instructions from the jsonl files.
• Benchmark the performance of reinforcement learning agents on the defined long-horizon task_ABC_D structure.
• Develop and evaluate models for learning language-conditioned manipulation policies from the multimodal video and annotation data.

Strengths

• Dataset is associated with a peer-reviewed publication in IEEE RA-L, providing a formal benchmark.
• Provides multimodal data streams including video and structured annotations for language-conditioned tasks.

Limitations

• Specific dataset size, row count, and detailed column schema are not provided in the input.
• The input lacks information on sample diversity, potential sensor noise, or action label quality.

Provenance

Source

Research by Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard.

Collection Method

Created as a benchmark for language-conditioned policy learning, likely involving robot teleoperation or simulation.

Freshness

The dataset record on the platform was last updated on 2025-10-11.