Multimodal Mind2Web: Web Screenshots Aligned with HTML for Agent Training

Multimodal-Mind2Web aligns HTML documents from the Mind2Web dataset with their corresponding webpage screenshot images. The dataset was created by osunlp to address the inconvenience of loading images from the original ~300GB raw dump and was last updated on June 5, 2024.

Use Cases

• Training multimodal agents to understand web interfaces based on aligned HTML and screenshot data.
• Evaluating agent performance on complex, real-world web tasks based on the dataset's instruction-following structure.
• Developing models that can parse visual and structural webpage information for navigation and interaction.

Strengths

• Addresses a specific data accessibility issue by providing pre-aligned screenshots for the Mind2Web dataset.
• Derived from the Mind2Web raw dump, which is designed for complex, real-world web tasks.

Limitations

• Description metadata is limited; actual data quality requires manual inspection after download.
• Column-level documentation is absent; field semantics must be inferred after download.
• Row count and file formats are unknown, which may limit suitability assessment.

Provenance

Source

huggingface, author osunlp

Collection Method

Aligned HTML documents from the Mind2Web raw dump with corresponding webpage screenshots.

Time Range

null

Freshness

Last updated 2024-06-05 05:12:21; freshness should be verified.

Geography

null