MMAU-Pro Benchmark for Multimodal Audio Intelligence Evaluation

MMAU-Pro is a benchmark dataset for evaluating audio intelligence in multimodal models, covering speech, environmental sounds, and music. It contains 5,305 expert-annotated question–answer pairs with audios sourced from the wild. The dataset was created by gamma-lab-umd and updated in August 2025.

Use Cases

• Benchmark model performance on 49 distinct perceptual and reasoning skills using expert-annotated question–answer pairs.
• Train models for audio question-answering tasks using the annotated audio and text pairs.
• Evaluate multimodal reasoning on combinations of speech, environmental sounds, and music.
• Assess model generalization on novel challenges introduced by the dataset.

Strengths

• 5,305 expert-annotated question–answer pairs.
• Covers 49 distinct perceptual and reasoning skills across multiple audio domains.

Limitations

• Dataset size is relatively small for training large models, with only 5,305 samples.
• Specific details on audio duration, class balance, and geographic source bias are not provided.

Provenance

Source

gamma-lab-umd via Hugging Face.

Collection Method

Audios sourced directly from the wild and annotated by experts.

Freshness

Last updated on 2025-08-28.