SB-Bench: Stereotype Bias Benchmark for Large Multimodal Models

A benchmark dataset comprising over 14,500 questions on non-synthetic images, created to assess stereotype biases in Large Multimodal Models (LMMs). The dataset, authored by ucf-crcv, was last updated on May 16, 2025. It spans nine diverse domains and 54 sub-domains to rigorously evaluate LMM performance in visually grounded stereotypical scenarios.

Use Cases

• Benchmarking stereotype bias in LMMs based on the described 14.5k+ visual questions.
• Evaluating model fairness across diverse domains based on the nine domains and 54 sub-domains mentioned.
• Training or fine-tuning models to reduce harmful societal prejudices based on the curated non-synthetic images.
• Researching the intersection of computer vision and social bias based on the described visually grounded stereotypical scenarios.

Strengths

• Over 14,500 questions provide a substantial testbed for evaluation.
• Covers nine diverse domains and 54 sub-domains for broad assessment.
• Uses carefully curated non-synthetic images, suggesting real-world relevance.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

ucf-crcv

Collection Method

Likely manually curated for bias assessment, but specific collection method is not detailed.

Freshness

Last updated 2025-05-16 08:27:48; freshness should be verified.