AssistantBench: Evaluating Web Agents on Realistic Tasks

AssistantBench is a dataset for evaluating web agents on realistic and time-consuming tasks, as described in the 2024 arXiv paper 'AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?'. The dataset was uploaded to Hugging Face on 2024-07 26. The authors are Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant.

Use Cases

• Benchmarking the performance of web agents based on the realistic tasks described in the paper.
• Training web agents to handle time-consuming, multi-step interactions based on the task structure.
• Analyzing failure modes of web agents based on the evaluation framework.
• Comparing different agent architectures or prompting strategies based on the benchmark tasks.

Strengths

• Dataset is directly linked to a peer-reviewed arXiv paper (2407.15711) published in 2024.
• Platform tags indicate a specific focus on question-answering tasks for web agents.
• Last updated on 2024-07-26, suggesting recent maintenance.

Limitations

• Description metadata is limited; actual data quality requires manual inspection after download.
• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.

Provenance

Source

Hugging Face, linked to the arXiv paper 'AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?'.

Collection Method

Likely created for the research described in the associated arXiv paper.

Time Range

Paper published in 2024.

Freshness

Last updated 2024-07-26 13:00:40.

Geography

Region tag indicates 'us', but specific geographic coverage is unknown.