VideoChat2 Instruction-Tuning Data for Multimodal AI

1.9 million data annotations in JSON format, created by OpenGVLab for instruction-tuning multimodal AI models. The dataset was last updated in June 2024. It is derived from image datasets like M3IT, which underwent quality filtering processes.

Use Cases

• Training models to generate text responses based on visual inputs using instruction-annotation pairs.
• Fine-tuning visual question answering models using the provided question-answer annotations.
• Benchmarking model performance on multimodal instruction-following tasks using the structured JSON annotations.
• Studying the effect of data quality filtering (e.g., typo correction, rephrasing) on model training outcomes.

Strengths

• Contains 1.9 million data annotations, providing a substantial scale for training.
• Data quality was improved by filtering lower-quality entries and correcting typos from source datasets.

Limitations

• The core dataset provides only JSON annotations; corresponding images and videos must be sourced separately, adding complexity.
• Specific row counts, column details, and sample data are unavailable, limiting initial assessment.

Provenance

Source

OpenGVLab, derived from image datasets including M3IT.

Collection Method

Annotations created for instruction-tuning; source data filtered for quality (typo correction, rephrasing).

Freshness

Last updated on 2024-06-29.

Geography

Region tags include 'us', but specific spatial coverage is unknown.