Written and Spoken Digits Database for Multimodal Fusion

A multimodal dataset of 70,000 samples constructed by pairing handwritten digit images with spoken digit audio clips. The handwritten data is sourced from the MNIST database, and the spoken data is extracted from the Google Speech Commands dataset, with audio pre-processed into Mel Frequency Cepstral Coefficients. The dataset was created by Lyes Khacef and colleagues for research in multimodal fusion.

Use Cases

• Train multimodal neural networks for joint representation of visual and auditory data based on the paired image and audio features.
• Benchmark cross-modal retrieval systems using the associated written and spoken digit samples.
• Develop models for audio-visual synchronization or generation tasks based on the aligned digit class labels.
• Research multimodal self-supervised learning techniques using the standardized and normalized MFCC and image features.

Strengths

• Provides 70,000 aligned multimodal samples (60,000 training, 10,000 test) for direct use.
• Uses established source datasets: MNIST for images and Google Speech Commands for audio.
• Audio features are pre-processed into standardized 507-dimensional MFCC vectors.
• Maintains the original train/test partitioning from the source datasets.

Limitations

• Spoken digit samples were duplicated to match the image count, which may introduce redundancy.
• Column-level documentation is absent; field semantics must be inferred after download.
• Last update date is unknown; freshness unverified.

Provenance

Source

Constructed from MNIST handwritten digits database and Google Speech Commands dataset.

Collection Method

Digits of the same class were associated, respecting original dataset partitions, with audio pre-processed via MFCC extraction.