NLLB: No Language Left Behind Parallel Text for 1,613 Language Pairs

148 English-centric and 1,465 non-English-centric language pairs of parallel text mined by Meta AI using the stopes library and LASER3 encoders. The complete dataset is approximately 450GB in size and was released by AllenAI in 2022. It is based on metadata for mined bitext and supersedes previous CCMatrix versions.

Use Cases

• Train machine translation models based on the 1,613 language pairs.
• Benchmark translation quality across diverse language families based on the mined bitext.
• Develop cross-lingual language understanding models based on the parallel sentences.
• Study data mining techniques for low-resource languages based on the LASER3 encoder methodology.

Strengths

• Contains parallel text for 1,613 language pairs, including many low-resource languages.
• The complete dataset is approximately 450GB, indicating a substantial scale.
• Uses a documented mining pipeline (stopes library, LASER3 encoders) for reproducibility.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Last updated 2022-09-29 18:53:15; freshness should be verified.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

Meta AI (mined bitext metadata), hosted by AllenAI.

Collection Method

Mined using the stopes library and LASER3 encoders.

Freshness

Last updated 2022-09-29 18:53:15.