GhostUI: 1,000+ Hidden Gesture Interactions in Mobile Applications

GhostUI contains between 1,000 and 10,000 records of mobile application interfaces focusing on hidden gesture-based interactions like swipes and long presses. Developed by the GhostUI team for CHI 2026, it maps UI screenshots to interactions that lack explicit visual cues. The data supports image-to-text tasks for identifying non-obvious user interface elements and was last updated in March 2026.

Use Cases

• Training image-to-text models to generate descriptions for 'swipe' and 'long press' interactions from static UI screenshots
• Developing autonomous mobile agents that can navigate interfaces using 'double tap' gestures that lack visible cues
• Benchmarking UI accessibility tools to detect 'hidden interactions' that do not have corresponding visual affordances

Strengths

• Includes 1,000 to 10,000 annotated records
• Targets specific gesture types including swipe, long press, and double tap
• Released under the MIT license for open research use

Limitations

• Small sample size of under 10,000 records compared to general UI datasets
• Geographic bias toward US-based mobile applications as indicated by metadata tags

Provenance

Source

GhostUI Research Team (Arxiv 260119258)

Freshness

Last updated March 2026

Geography

United States