Wikipedia: Cleaned Full-Text Articles in Multiple Languages

A multilingual text corpus built from Wikipedia dumps, with one split per language. Each example contains the cleaned content of one full Wikipedia article, with markdown and unwanted sections like references stripped. The dataset was created by openskyml and was last updated on the platform in October 2023.

Use Cases

• Train multilingual language models based on the cleaned, full-text article content.
• Benchmark cross-lingual information retrieval systems using the structured knowledge in the articles.
• Develop text summarization models based on the long-form article format.
• Analyze linguistic patterns and knowledge representation across different languages.

Strengths

• Data is sourced from the authoritative Wikipedia dump archive.
• Content is cleaned to remove markdown and unwanted sections like references.
• The dataset is structured with one split per language, facilitating language-specific analysis.

Limitations

• Description metadata is limited; actual data quality requires manual inspection after download.
• Last updated 2023-10-08 10:37:06; freshness should be verified.
• Column-level documentation is absent; field semantics must be inferred after download.

Provenance

Source

Wikipedia dumps (https://dumps.wikimedia.org/)

Collection Method

Built from dumps, with cleaning applied to strip markdown and unwanted sections.

Freshness

Last updated 2023-10-08 10:37:06.