PixelProse: 16.9 Million Dense Synthetic Image Captions

PixelProse contains 16,896,214 image-caption pairs featuring dense synthetic descriptions generated by Gemini 1.0 Pro Vision. Released in 2024 by researchers at the University of Maryland (tomg-group-umd), the collection provides detailed textual representations for images sourced from CommonPool and CC12M.

Use Cases

• Fine-tuning text-to-image generators using the dense captions to improve prompt alignment
• Training vision-language models for image-to-text tasks using the synthetic descriptions
• Evaluating visual question answering models against detailed scene narratives

Strengths

• 16.9 million total records
• Captions generated by Gemini 1.0 Pro Vision for high descriptive detail
• Openly licensed under CC BY 4.0

Limitations

• Synthetic captions may contain model-specific hallucinations or biases from the Gemini 1.0 Pro Vision model
• Dependency on external image sources like CC12M which are subject to link rot and availability issues

Provenance

Source

University of Maryland (tomg-group-umd) via arXiv:2406.10328

Collection Method

Synthetic generation using Gemini 1.0 Pro Vision on images from CommonPool and CC12M

Freshness

Last updated December 2024

Geography

Global (web-scraped images)