Multimodal Robot Trajectories with Natural Language Instructions

SandGO Dataset is a multimodal resource for embodied AI and robot learning, containing approximately 168 long-horizon trajectory episodes and 38,011 time steps. It was created by XDUImageLab from refined MarsMind_data to support tasks like long-sequence decision-making and instruction following. The dataset was last updated in March 2026.

Use Cases

• Train Vision-Language-Action models to predict robot actions from natural language instructions and multi-view RGB images.
• Develop policies for long-horizon tasks like navigation and object manipulation using sequences of robot states and actions.
• Benchmark multimodal perception models by fusing data from LiDAR point clouds, depth point clouds, and RGB camera feeds.
• Study instruction grounding by correlating natural language commands for tasks like 'grasp' or 'place' with the corresponding robot state-action sequences.

Strengths

• Contains 168 long-sequence trajectory episodes for training on extended tasks.
• Provides 38,011 individual time steps of synchronized multimodal data.
• Includes multiple sensor modalities: main camera RGB, wrist camera RGB, LiDAR point clouds, and depth point clouds.

Limitations

• The dataset size of 168 episodes may be limited for training large-scale models without augmentation.
• Geographic and environmental coverage is unspecified, which may limit model generalization.
• Potential class or task imbalance is not detailed across the listed activities like cruising, obstacle avoidance, and grasping.

Provenance

Source

XDUImageLab, derived from refined MarsMind_data.

Collection Method

Processed through refinement, deduplication, and structuring of raw robot interaction data.

Freshness

Last updated March 2026.