Multimodal Language Analysis Benchmark With 61,000 Samples

61,000+ multimodal samples across text, video, and audio modalities from nine datasets. The MMLA benchmark, created by THUIAR, includes data from films, TV series, YouTube, Vimeo, Bilibili, TED, and improvised scripts for evaluating foundation models.

Use Cases

• Evaluate foundation model performance on text, video, and audio modalities using the benchmark's structured tasks.
• Analyze multimodal alignment by comparing features extracted from the video and audio samples with associated text.
• Train models for real-world versus acted scenario classification using samples from films, TV series, YouTube, Vimeo, Bilibili, and TED.
• Benchmark model robustness across nine different source datasets aggregated in this collection.

Strengths

• Contains over 61,000 multimodal samples, providing substantial scale for evaluation.
• Aggregates data from nine distinct source datasets, increasing diversity.
• Covers three core modalities: text, video, and audio.
• Includes samples from both acting and real-world scenarios like films, TV series, and online platforms.

Limitations

• Specific column definitions, file formats, and sample data are not provided in the input, hindering immediate technical understanding.
• The dataset's composition and potential class imbalances across its nine sources are not detailed.

Provenance

Source

THUIAR, via Hugging Face.

Collection Method

Aggregated from nine existing datasets.

Time Range

null

Freshness

Last updated on August 13, -08-13.

Geography

null