Lemma-based frequency lists for languages including French, Japanese, Spanish, Italian, and Portuguese were generated from approximately 1GB of Netflix subtitle data. The lists consolidate inflected forms, counting 'run', 'runs', and 'running' as instances of the same lemma. The dataset was created by joshdavham and last updated on May 4, 2026.
Use Cases
- Build language-specific tokenizers or vocabulary lists based on lemma frequencies.
- Analyze linguistic patterns and common word usage across multiple languages.
- Pre-train or fine-tune multilingual NLP models using real-world subtitle text.
- Compare the relative prevalence of concepts or terms across different languages.
Strengths
- Focuses on lemmas, which groups inflected word forms for more meaningful frequency analysis.
- Derived from a real-world text source of approximately 1GB of Netflix subtitles.
- Includes multiple languages, specifically French, Japanese, Spanish, Italian, and Portuguese.
Limitations
- Column-level documentation is absent; field semantics must be inferred after download.
- Row count and file formats are unknown, which may limit suitability assessment.
- Data may reflect thematic and demographic bias inherent to the source of Netflix shows and films.
Provenance
- Source
- Subtitles scraped from Netflix shows and films.
- Collection Method
- Parsed using relevant spaCy models.
- Freshness
- Last updated 2026-05-04 02:42:19; freshness should be verified.