CUAVerifierBench: A Human-Annotated Benchmark for Computer-Using Agent Verifiers

CUAVerifierBench is an evaluation benchmark for verifiers of computer-using agents, created by Microsoft. It contains human-annotated trajectories of agent interactions to judge task completion. The dataset was last updated on 2026-04-21.

Use Cases

• Training verifier models based on human judgments of agent trajectories.
• Benchmarking the performance of different verifier architectures on a standardized task.
• Studying the failure modes of computer-using agents based on annotated incorrect trajectories.

Strengths

• Dataset is human-annotated, which likely provides a reliable ground truth for evaluation.
• Focuses on a specific and emerging evaluation need for computer-using agent verifiers.

Limitations

• Description metadata is limited; actual data quality requires manual inspection after download.
• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.

Provenance

Source

Microsoft

Collection Method

Human-annotated trajectories of agent interactions, as described in the associated 'Universal Verifier' paper.

Freshness

Last updated 2026-04-21 19:30:05; freshness should be verified.