Chug: Sharded Multi-Modal Loaders for PDF and Image-Text Data

Hugging Face released Chug in April 2024 to provide sharded dataset loaders and decoders for multi-modal document, image, and text data. It focuses on efficient distributed training using WebDataset and PDF formats for computer vision and document understanding tasks.

Use Cases

• Distributed training of multi-modal models using WebDataset shards
• Decoding PDF documents for document-understanding tasks
• Implementing minimal sharded loaders for computer vision pipelines

Strengths

• Apache-2.0 licensed for open commercial use
• Native support for WebDataset and PDF-document formats
• Optimized for distributed-training environments

Limitations

• Undefined record counts and specific metadata in the repository
• Functions primarily as a loader utility requiring external data sources

Provenance

Source

Hugging Face

Freshness

Last updated April 2024.