Video-Holmes: 1,837 Questions for MLLM Video Reasoning on 270 Short Films

Video-Holmes is a benchmark of 1,837 questions designed to evaluate the complex reasoning capabilities of Multimodal Large Language Models (MLLMs). The questions are derived from 270 manually annotated suspense short films, each ranging from 1 to 5 minutes in length. It was created by researchers from Tencent ARC Lab and City University of Hong Kong and was last updated on the Hugging Face platform in September 2025.

Use Cases

• Benchmarking MLLM performance on complex video reasoning based on manually annotated suspense films.
• Training models for temporal and causal reasoning based on short video narratives.
• Developing evaluation frameworks for multimodal AI based on a curated set of 1,837 questions.

Strengths

• Contains 1,837 questions for detailed evaluation.
• Based on 270 manually annotated short films, suggesting curated content.
• Films are specifically suspense genre, providing a focused domain for reasoning.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Data may reflect genre bias inherent to the selected suspense films.

Provenance

Source

TencentARC (ARC Lab, Tencent PCG) and City University of Hong Kong.

Collection Method

Questions derived from 270 manually annotated suspense short films.

Freshness

Last updated 2025-09-23 13:46:46; freshness should be verified.