Polaris: 100K+ Human Feedback Records for Image Captioning Evaluation

Polaris contains between 100,000 and 1,000,000 records of human feedback on image-caption pairs, released by researcher yuwd in 2024. This multimodal dataset supports the development of evaluation metrics that align with human judgment as described in the CVPR 2024 paper "Polos."

Use Cases

• Training multimodal metrics to evaluate image captioning models using human feedback scores
• Benchmarking image-to-text alignment by comparing model outputs against human judgments
• Fine-tuning vision-language models for better correlation with human preferences

Strengths

• Scale of 100,000 to 1,000,000 records
• Peer-reviewed at CVPR 2024
• BSD licensed for open research

Limitations

• Lack of explicit column documentation in the repository metadata
• Potential for label noise inherent in subjective human feedback for image captioning

Provenance

Source

yuwd, CVPR 2024 (Arxiv 2402.18091)

Collection Method

annotated

Freshness

Last updated March 2024; reflects state-of-the-art research for CVPR 2024.