BKAINewsCorpus: Vietnamese News Articles, 32 Million Records

Vietnamese news corpus containing approximately 32 million articles. The dataset was created by merging the Binhvq News Corpus of about 20 million articles with an additional 10 million articles collected up to November 2023. It was published by bkai-foundation-models and last updated on the platform in March 2024.

Use Cases

• Train large language models for Vietnamese based on the large-scale text corpus.
• Conduct temporal analysis of news topics based on articles spanning until late 2023.
• Perform text deduplication research using the fuzzy deduplication process mentioned in the description.
• Analyze linguistic patterns and media trends across diverse Vietnamese news sources.

Strengths

• Approximately 32 million articles provide a substantial volume of text for training.
• Integrates a previously established corpus with a significant update of 10 million newer articles.
• Articles are sourced from diverse outlets, which may offer varied perspectives.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• The description notes fuzzy deduplication was applied, but the specific method and remaining duplication level are unknown.
• Row count is an approximate figure; the exact number of unique articles is not specified.

Provenance

Source

Aggregated from diverse Vietnamese news sources, merging the Binhvq News Corpus with new collections.

Collection Method

Articles were gathered and integrated, followed by fuzzy deduplication.

Time Range

Articles up to November 2023; the original Binhvq corpus was last updated in May 2021.

Freshness

Last updated 2024-03-05 13:55:35; articles collected up to November 2023.

Geography

Vietnam (inferred from language and source description).