AMEX: Android Multi-annotation Expo for Mobile GUI Agents

AMEX provides multi-annotated Android GUI data for training mobile agents, released in 2024 by Yuxiang007. The dataset includes screenshots paired with instruction-action chains and fine-grained element metadata such as bounding boxes and functional descriptions.

Use Cases

• Training mobile GUI agents to execute tasks using instruction_anno action chains
• UI element detection and localization using element_anno bounding boxes
• Generating natural language descriptions of mobile interfaces using screenshot description fields

Strengths

• Includes bounding box coordinates for UI element localization
• Provides sequential GUI-action chains for multi-step task instructions
• Multi-modal structure linking visual screenshots to functional element descriptions

Limitations

• The repository currently provides only a sample rather than the full dataset
• Lack of explicit metadata regarding Android version coverage or app category diversity

Provenance

Source

Arxiv paper 2407.17490: AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents

Collection Method

Annotated

Freshness

Last updated September 2024.