Multimodal Dataset with One Trillion Text Tokens and 3.4 Billion Images

MINT-1T is an open-source multimodal interleaved dataset containing 1 trillion text tokens and 3.4 billion images, a tenfold increase in scale compared to prior open collections. It was created by a team from the University of Washington and includes data from previously untapped sources like PDFs and arXiv papers. The dataset was uploaded to the platform in September 2024.

Use Cases

• Pretraining multimodal large language models on interleaved sequences of text tokens and images.
• Studying the scaling laws of model performance using 1 trillion text tokens and 3.4 billion images.
• Training models to understand content from academic sources using the included PDF and arXiv paper data.
• Benchmarking image-to-text generation capabilities on a dataset containing billions of image-text pairs.

Strengths

• Contains 1 trillion text tokens, a 10x scale-up from existing open-source datasets.
• Includes 3.4 billion images for multimodal training.
• Incorporates data from diverse sources like PDFs and arXiv papers.

Limitations

• Specific data quality metrics, class balance, or geographic diversity are not detailed in the provided description.
• The immense scale may present significant computational and storage challenges for many users.
• The exact composition, preprocessing steps, and potential duplicates within the 1 trillion tokens are unknown.

Provenance

Source

University of Washington research team (mlfoundations).

Collection Method

Aggregated from multiple open sources, including PDFs and arXiv papers.

Time Range

Includes a snapshot from June 2023, as indicated in the title.

Freshness

Dataset page was last updated in September 2024.

Geography

null