OSWorld Benchmark for Multimodal Agents in Virtual Machines

OSWorld provides task examples, retrieval documents, and virtual machine snapshots for benchmarking multimodal agents performing open-ended tasks in real computer environments. The dataset was created by xlangai and last updated in October 2024. It supports evaluation on both x86 and arm64 machine architectures using VMware or VirtualBox.

Use Cases

• Benchmarking agent performance on specific tasks using the provided task examples and evaluation_examples.zip.
• Training retrieval-augmented agents using the included retrieval documents.
• Testing agent generalization across different machine architectures (x86, arm64) via the virtual machine snapshots.
• Evaluating multimodal reasoning in real computer environments using the VMware/VirtualBox snapshots.

Strengths

• Includes virtual machine snapshots for two major architectures (x86, arm64).
• Provides specific task examples and retrieval documents for benchmarking.

Limitations

• Unknown dataset size, row count, and specific column structure.
• Relies on external virtualization software (VMware/VirtualBox) for use, which adds complexity.

Provenance

Source

xlangai

Collection Method

Created for the OSWorld benchmark project, includes curated task examples and VM snapshots.

Freshness

Last updated October 2024.