PySpark Huggingface: Official Spark Connector for Distributed Data

Hugging Face developed this PySpark custom data source, updated in August 2025, to facilitate the integration of the thousands of datasets on the Hugging Face Hub into Spark environments. It provides a programmatic interface to stream or load diverse data modalities directly into Spark DataFrames for distributed processing.

Use Cases

• Distributed feature engineering on 'text' or 'image' columns using Spark's parallel processing.
• Large-scale data validation of Hub records by applying Spark SQL schemas to 'label' or 'feature' columns.
• ETL workflows that join Hub datasets with internal data using shared 'id' or 'timestamp' keys.

Strengths

• Official implementation by Hugging Face
• Apache-2.0 license
• Supports Spark DataSource V2 API

Limitations

• Does not contain static data records itself
• Requires active network connection to fetch Hub datasets
• Performance is limited by Hugging Face Hub API rate limits

Provenance

Source

Hugging Face GitHub repository

Collection Method

Software development

Freshness

Updated as of August 2025.

Geography

Global