Osworld G: Computer-Use Grounding via UI Synthesis

Osworld G provides a benchmark for computer-use grounding through UI decomposition and synthesis, released by xlang-ai as a NeurIPS 2025 Spotlight. It facilitates the training of Large Action Models (LAMs) by generating multimodal data that pairs visual GUI elements with natural language grounding instructions.

Use Cases

• Training Vision-Language Models (VLMs) to identify specific UI elements using the provided grounding labels
• Benchmarking Large Action Models (LAMs) on their ability to perform grounding across diverse GUI layouts and screenshots
• Developing RPA systems that utilize UI decomposition metadata to navigate complex software interfaces

Strengths

• NeurIPS 2025 Spotlight peer-reviewed status
• Scalable UI decomposition methodology for data generation
• Multimodal alignment between visual GUI elements and text instructions

Limitations

• Reliance on synthetic data generation may introduce distribution shifts compared to organic human-computer interaction
• Lack of explicit row counts in the initial release metadata

Provenance

Source

xlang-ai

Collection Method

UI decomposition and synthesis

Freshness

Last updated November 2025.