Uncopyrighted Text Corpus for Language Model Training

August 2023 dataset derived from The Pile, with copyrighted content removed to respect author rights. Created by monology, this collection is intended for training large language models without copyright infringement. The methodology involved removing content from specific copyrighted sources like Books3 and BookCorpus2.

Use Cases

• Train language models on text data filtered for copyright compliance.
• Benchmark model performance on a corpus free from copyrighted Books3 and BookCorpus2 content.
• Analyze the linguistic characteristics of a text corpus curated from non-copyrighted sources.
• Fine-tune text generation models using the OpenSubtitles and YTSubtitles portions of the cleaned data.

Strengths

• Derived from The Pile, a known large-scale text corpus.
• Explicitly removes content from 5 specific copyrighted source datasets (Books3, BookCorpus2, OpenSubtitles, YTSubtitles, OWT2).
• Created in August 2023 with a focus on modern copyright concerns for AI training.

Limitations

• Exact size, row count, and specific remaining data sources are unknown.
• The cleaning methodology may introduce bias by excluding entire domains of copyrighted text.
• Lacks documentation on the license, file formats, and sample data for verification.

Provenance

Source

Derived from The Pile dataset.

Collection Method

Created by removing all content from the Books3, BookCorpus2, OpenSubtitles, YTSubtitles, and OWT2 subsets of The Pile.

Freshness

Last updated on 2023-08-31.