T2I-CompBench: Compositional Text-to-Image Evaluation Benchmark

Karine-Huang developed T2I-CompBench in 2023 to provide a standardized framework for evaluating compositionality in text-to-image models. The benchmark includes curated text prompts and evaluation metrics published in NeurIPS 2023 and TPAMI to assess how models handle complex textual instructions.

Use Cases

• Measuring attribute binding accuracy by testing prompts with multiple color-object pairs
• Evaluating spatial reasoning using prompts that specify relative positions of objects
• Benchmarking text-to-image model consistency across complex multi-entity descriptions

Strengths

• Peer-reviewed methodology published in NeurIPS 2023 and TPAMI
• Categorized prompts targeting specific compositional failures like attribute binding
• Open-source MIT license allowing for broad research application

Limitations

• Does not include pre-generated images, requiring users to perform their own model inference
• Focuses strictly on compositionality rather than general image aesthetic or resolution quality

Provenance

Source

Karine-Huang, NeurIPS 2023 and TPAMI publication

Collection Method

Expert-curated prompts and evaluation metrics

Freshness

Last updated in late 2024, reflecting evaluation standards established in the 2023 NeurIPS and TPAMI publications.