Wikipedia Full: Cleaned Multilingual Article Corpus

Wikimedia Wikipedia provides cleaned articles from all language editions of Wikipedia. The dataset is built from Wikipedia dumps, with one subset per language, each containing a single train split. Each example contains the content of one full article, cleaned to strip markdown and unwanted sections like references.

Use Cases

• Train large language models on factual, long-form text based on the cleaned article content.
• Build multilingual search or retrieval systems based on the cross-lingual knowledge base.
• Conduct linguistic analysis or study knowledge representation across languages using the parallel language subsets.
• Fine-tune models for tasks like summarization or question answering using the structured article text.

Strengths

• Contains articles from all language editions of Wikipedia, indicating broad multilingual coverage.
• Data is cleaned to strip markdown and unwanted sections like references, suggesting a focus on core article text.
• Each language subset is pre-processed from a recent dump, as noted in the description.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment for large-scale projects.
• Description metadata is limited; actual data quality and completeness require manual inspection.

Provenance

Source

Wikipedia dumps (https://dumps.wikimedia.org/)

Collection Method

Built from dumps, with cleaning applied to strip markdown and unwanted sections.

Freshness

Last updated 2025-12-24 00:55:47; freshness should be verified.