Taiwan Text Excellence 2B: High-Quality Traditional Chinese News and Articles

A corpus of approximately 2 billion tokens of Traditional Chinese text, processed for language model performance. The dataset, created by liswei and last updated on June 2, 2024, contains high-quality news and articles focused on Taiwanese culture. It has been de-duplicated and cleaned using rule-based and learning-based filters.

Use Cases

• Fine-tune language models for Traditional Chinese based on the high-quality news and article corpus.
• Train models for Taiwanese cultural understanding based on the described content focus.
• Benchmark text cleaning and normalization pipelines based on the described rule-based and learning-based filters.
• Study linguistic patterns in Taiwanese media based on the described news and article sources.

Strengths

• Contains approximately 2 billion tokens, providing substantial scale for model training.
• Data processing pipeline is explicitly optimized for LLM performance.
• Underwent de-duplication and cleaning with both rule-based and learning-based filters.
• Specific cleaning steps mentioned include normalization of URLs, emails, HTML tags, abnormal characters, and number formats.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count and file formats are unknown, which may limit suitability assessment.
• Data may reflect geographic and source bias inherent to its news and article collection focused on Taiwan.

Provenance

Source

huggingface

Collection Method

Aggregated from high-quality news and articles; processing pipeline details are described.

Time Range

null

Freshness

Last updated 2024-06-02 07:45:23; freshness should be verified.

Geography

Taiwan