Img2Dataset: High-Speed Image Downloader for 100M+ URL Records

Img2Dataset is a high-performance utility created by rom1504 for converting massive URL collections into structured image datasets. It can download, resize, and package 100 million URLs in 20 hours on a single machine, facilitating the creation of large-scale multimodal repositories.

Use Cases

• Downloading and resizing images from URL columns for computer vision training
• Packaging image-text pairs into WebDataset shards for multimodal model development
• Converting large-scale metadata files into local image repositories for offline analysis

Strengths

• Processes 100 million URLs in 20 hours
• Automated resizing to specific dimensions
• Supports WebDataset packaging for distributed training

Limitations

• Final dataset quality depends on external URL availability
• Requires high network bandwidth for large-scale operations
• Does not provide the raw images directly

Provenance

Source

GitHub repository by rom1504

Collection Method

scraped

Freshness

Last updated October 2025

Geography

global