MMSkills: Multimodal Procedure Data for GUI Task Automation

34 skills for the Chrome domain and 26 for GIMP are included in this dataset from the Towards MMSkills project. It contains procedure descriptions, runtime state cards, and visual references organized by application domain. The dataset was authored by zhangkangning and last updated on 2026-05-11.

Use Cases

• Train multimodal agents to execute GUI tasks based on the structured SKILL.md procedures.
• Benchmark AI performance on real-world software workflows using the provided runtime_state_cards.json.
• Develop visual grounding models for GUI elements using the visual references under Images/.
• Analyze task complexity and state transitions using the provided statistics on cards and views.

Strengths

• Includes 134 state cards for the Chrome domain, providing detailed runtime context.
• Organizes skills by application domain (e.g., Chrome, GIMP) for structured evaluation.
• Provides transition card statistics (e.g., 24 for Chrome, 17.9%) for analyzing task flow.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Freshness should be verified as the last update timestamp is 2026-05-11.

Provenance

Source

Towards MMSkills project, hosted on Hugging Face by author zhangkangning.

Collection Method

Likely collected and structured as part of the OSWorld Ubuntu MMSkill package.

Freshness

Last updated 2026-05-11 13:12:35.