Grounded visual computer-use trajectories generated by SynthUX simulate worker sessions within a simulated company. Each record captures a worker's device session, where a goal expands into a node tree and terminal nodes drive real desktop-simulator applications. The observed trajectory includes screen video and per-node frames, recorded by author jacob-valdez and last updated on June 3, 2026.
Use Cases
- Train AI agents for desktop automation based on low-level mouse/keyboard input sequences.
- Analyze human task decomposition patterns based on goal-expansion into node trees.
- Develop multimodal models for screen understanding based on recorded screen video and per-node frames.
- Benchmark human-computer interaction in simulated office environments based on trajectories from applications like Terminal, VS Code, Browser, and Slack.
Strengths
- Trajectories include screen video, providing a visual record of interactions.
- Data generation is grounded within a simulated company environment, offering contextual realism.
- Low-level input recording captures mouse and keyboard actions driving applications like Terminal, Notes, VS Code, Browser, Slack, Mail, Sheets, and Slides.
Limitations
- Column-level documentation is absent; field semantics must be inferred after download.
- Row count is unknown, which may limit suitability assessment.
- Description metadata is limited; actual data quality requires manual inspection after download.
Provenance
- Source
- huggingface
- Collection Method
- Generated by SynthUX within a simulated company environment.
- Freshness
- Last updated 2026-06-03 17:14:55; freshness should be verified.