ShapeCodeBench Eval V1: 150 Synthetic Images for Multimodal Program Reconstruction

150 grayscale 512x512 PNG images form a frozen evaluation split for ShapeCodeBench. This synthetic benchmark tests if multimodal models can reconstruct executable drawing programs from rendered shape images, with 50 easy, 50 medium, and 50 hard examples. The dataset was created by author shivamk3r and last updated on Hugging Face in May 2026.

Use Cases

• Benchmarking multimodal model performance on program reconstruction tasks based on rendered shape images.
• Training models to infer executable drawing programs from visual inputs based on the ShapeCodeBench DSL.
• Evaluating the difficulty scaling of visual reasoning tasks using the easy, medium, and hard example splits.

Strengths

• Contains 150 precisely defined evaluation examples, enabling standardized benchmarking.
• Provides a difficulty gradient with 50 easy, 50 medium, and 50 hard examples for nuanced assessment.
• Includes canonical ground-truth programs and generation metadata for each image, facilitating detailed analysis.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• The dataset is synthetic, which may limit its generalizability to real-world visual program synthesis tasks.

Provenance

Source

huggingface

Collection Method

Synthetically generated for benchmark creation.

Freshness

Last updated 2026-05-15 13:43:52; freshness should be verified.