MPCI-Bench: Multimodal Pairwise Contextual Integrity Evaluation for Language-Model Agents

MPCI-Bench is a benchmark for evaluating the contextual integrity of multimodal language-model agents. Each benchmark pair starts from a VISPR image and contains two contrastive information flows: one appropriate case and one inappropriate case, each represented at three levels of increasing context. The dataset was created by Soojuu and was last updated on Hugging Face in May 2026.

Use Cases

• Benchmarking language-model agents' ability to discern appropriate from inappropriate information flows based on the described contrastive pairs.
• Evaluating AI models on privacy-utility trade-offs within realistic social narratives as described in the 'Story' level context.
• Training or fine-tuning models for contextual integrity using structured parameters from the 'Seed' level context.
• Researching multimodal reasoning where image context (VISPR) informs narrative appropriateness.

Strengths

• Provides structured evaluation with contrastive pairs (appropriate vs. inappropriate) for clear benchmarking.
• Offers context at three increasing levels (Seed, Story, etc.), allowing for nuanced analysis.
• Based on VISPR images, suggesting a multimodal foundation for the benchmark.

Limitations

• Description metadata is limited; actual data quality requires manual inspection after download.
• Column-level documentation is absent; field semantics must be inferred after download.
• Row count, file formats, and license information are unknown, which may limit suitability assessment.

Provenance

Source

Hugging Face dataset uploaded by author Soojuu.

Collection Method

Likely constructed for benchmarking purposes, starting from VISPR images and creating contrastive narrative cases.

Freshness

Last updated 2026-05-06 06:46:10; freshness should be verified.