4,000,000 image-caption pairs stored in PyArrow IPC format for high-performance multimodal training. The dataset utilizes memory-mapped files to enable low-latency data access during large-scale model optimization.
Use Cases
- Train multimodal encoders using the image and caption features
- Benchmark I/O performance using the pa.memory_map and pa.ipc.open_file methods
- Fine-tune generative models on 4 million text-image pairs
Strengths
- 4,000,000 image-caption pairs
- Distributed as PyArrow IPC files for memory-mapped reading
- Includes a PyTorch Dataset class template for immediate integration