Openjudge

A benchmark dataset for evaluating graders across text, multimodal, and agent scenarios. It supports the OpenJudge framework with labeled preference pairs for quality-assured grader development. The dataset was created by agentscope-ai and last updated on March 4, —.

Use Cases

• Benchmarking grader performance based on labeled preference pairs mentioned in the description
• Developing quality-assured graders for agent scenarios based on the described task categories
• Evaluating multimodal model coherence based on the image_coherence benchmark
• Testing grader capabilities on tool-use tasks based on the described tool category

Strengths

• Includes 12 agent evaluation tasks with 166 samples
• Contains 4 multimodal evaluation benchmarks with 80 samples
• Provides labeled preference pairs for structured grader development

Limitations

• Column-level documentation is absent; field semantics must be inferred after download
• Row count is unknown, which may limit suitability assessment
• Description metadata is limited; actual data quality requires manual inspection after download

Provenance

Source

agentscope-ai

Collection Method

Likely curated for the OpenJudge framework.

Time Range

null

Freshness

Last updated 2026-03-04 12:41:38; freshness should be verified

Geography

null