IDL: Industry Documents Library with 19 Million Pages and OCR

The Industry Documents Library (IDL) is a multimodal dataset containing 19 million pages of documents filtered from the UCSF library. Each document is available as a PDF, a TIFF image, a JSON file with Textract OCR annotations, and an older .ocr file. The dataset was created by pixparse and last updated on March 29, 2024.

Use Cases

• Train or evaluate Optical Character Recognition (OCR) models based on the extensive Textract annotations.
• Analyze document structure and layout using the provided PDF and image pairs.
• Conduct historical or industry-specific text mining on the large corpus of documents.
• Compare OCR performance between modern (Textract) and legacy systems using the dual annotation files.

Strengths

• Contains 19 million pages, indicating a large scale.
• Provides multiple representations (PDF, TIFF, JSON, .ocr) for each document, allowing for flexible analysis.
• Includes extensive Textract OCR annotations for each page.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

Filtered from the University of California, San Francisco (UCSF) Industry Documents Library.

Collection Method

Filtering and processing by the idl_data project.

Freshness

Last updated 2024-03-29 17:04:45; freshness should be verified.