Human Web Trajectories with Screenshots and Agent Actions

A dataset of human-collected web trajectories pairing instructions with sequences of webpage screenshots and corresponding agent actions like clicks, typing, and scrolling. It was created by AllenAI and last updated in March 2026. The dataset includes image and text modalities and is tagged as containing between 10,000 and 100,000 examples.

Use Cases

• Train multimodal agents to predict agent actions from sequences of webpage screenshots and instructions.
• Analyze patterns in human web navigation by examining the sequence of clicks, typing, and scrolling actions.
• Develop models that generate synthetic web trajectories by learning from the pairing of instructions and image sequences.
• Benchmark instruction-following agents on tasks requiring interaction with web interfaces via screenshots and actions.

Strengths

• Contains human-collected trajectories, providing realistic examples of web interaction.
• Includes multimodal data with both image sequences (screenshots) and text instructions.
• Dataset scale is indicated by tags suggesting a size between 10,000 and 100,000 examples.
• Maintained by a recognized research organization (AllenAI) with a recent update in March 2026.

Limitations

• Exact row count, column structure, and file size are unknown, hindering precise resource planning.
• Data format details are unclear, requiring users to consult the external dataset page for full specifications.
• Potential geographic bias is suggested by a 'Regionus' tag, indicating a possible focus on US-based web interactions.

Provenance

Source

AllenAI via Hugging Face.

Collection Method

Human-collected web trajectories.

Freshness

Last updated on 2026-03-24.

Geography

Potential US focus, as suggested by a 'Regionus' tag.