ScreenSpot: GUI Grounding Benchmark for Multimodal Models

ScreenSpot is an evaluation benchmark for GUI grounding created by researchers at Nanjing University and Shanghai AI Laboratory. It comprises over 1200 instructions from iOS, Android, macOS, Windows, and Web environments, along with annotations. The dataset was last updated on 2024-04-10.

Use Cases

• Benchmarking model performance on GUI grounding tasks based on text-based instructions.
• Training models to understand and interact with cross-platform UI elements based on annotated examples.
• Evaluating the generalization of multimodal models across different operating systems and web environments.

Strengths

• Over 1200 instructions provide a substantial test set for evaluation.
• Covers five distinct GUI environments: iOS, Android, macOS, Windows, and Web.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

Created by researchers at Nanjing University and Shanghai AI Laboratory.

Collection Method

Likely contains annotated instructions and corresponding GUI screens for evaluation.

Freshness

Last updated 2024-04-10 19:52:26; freshness should be verified.