VLM-SubtleBench: 10,000+ Image Pairs for Subtle Comparative Reasoning

VLM-SubtleBench provides between 10,000 and 100,000 image pairs to evaluate the subtle comparative reasoning capabilities of Vision-Language Models. Developed by KRAFTON and released in early 2026, the dataset targets domains where visual differences are nuanced, such as medical imaging and industrial anomaly detection.

Use Cases

• Evaluating VLM performance on industrial anomaly detection using the image-to-text comparative descriptions
• Testing multi-image reasoning capabilities by comparing visually similar medical imaging pairs
• Benchmarking visual question answering accuracy for subtle changes in aerial surveillance imagery

Strengths

• Scale of 10,000 to 100,000 records
• Focus on subtle visual differences rather than salient ones
• Covers specialized domains like medical and industrial imaging

Limitations

• Restricted to non-commercial use via CC BY-NC 4.0 license
• Lack of specific column metadata in the public repository description

Provenance

Source

KRAFTON

Freshness

Last updated March 2026.

Geography

United States