Multimodal Instruction Dataset With 721K Text-Image Examples

Mantis-Instruct contains 721,000 instruction tuning examples across 14 specialized subsets. It is a fully interleaved text-image dataset designed for training multimodal models on skills like co-reference, reasoning, and temporal understanding. The dataset was created by TIGER-Lab for training the Mantis model families.

Use Cases

• Train a multimodal model on 721K interleaved text-image examples to perform multi-image co-reference tasks.
• Fine-tune a vision-language model for temporal understanding using sequences of images and text from the dataset's subsets.
• Benchmark model performance on multi-image skills like comparing and reasoning covered by the 14 distinct subsets.

Strengths

• 721,000 instances provide substantial scale for model training.
• Covers 14 distinct subsets targeting specific multi-image skills.
• Fully interleaved text and image data supports complex multimodal instruction tuning.

Limitations

• Specific column structure and data format details are not provided in the input.
• Potential for skill or domain bias based on the composition of the 14 source subsets.

Provenance

Source

TIGER-Lab, via Hugging Face.

Collection Method

Aggregated from 14 subsets to cover multi-image skills.

Freshness

Last updated on December 25, 2024.