NLLB Primary Datasets Public Data Embedding: Multilingual Text for SONAR

A preprocessed collection of public data and NLLB-Seed data for generating embeddings with the SONAR model. The dataset excludes specific languages like Oromo (orm), Afar (aar), Ndebele (nde), and Venetian (ven) that are not supported by SONAR. The dataset was last updated on September 24, 2025, by the author jasonrichdarmawan on the Hugging Face platform.

Use Cases

• Train or evaluate multilingual text embedding models based on the preprocessed public data.
• Fine-tune language models for specific languages supported by the SONAR model, excluding the unsupported ones mentioned.
• Conduct cross-lingual similarity or retrieval tasks using the generated text embeddings.
• Benchmark the performance of the SONAR model on the included NLLB-Seed data subsets.

Strengths

• Dataset was last updated on 2025-09-24, indicating recent maintenance.
• Description specifies exclusions of languages unsupported by the SONAR model, clarifying scope.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count, file formats, and license information are unknown, which may limit suitability assessment.
• Data may reflect source bias inherent to the original public data and NLLB-Seed compilation.

Provenance

Source

huggingface

Collection Method

Preprocessing of public data and NLLB-Seed data.

Freshness

Last updated 2025-09-24 11:05:19; freshness should be verified.