Pre-built FM-indexes constructed with the Burrows–Wheeler Transform and suffix arrays using infini-gram-mini (Liu et al. 2025) over the ruggsea/infini-news-corpus. The indexes enable exact substring counting and document retrieval and power a live search API. The dataset was authored by ruggsea and last updated on July 1, 2026.
Use Cases
- Perform full-text search over a news corpus based on the described FM-index structure.
- Conduct exact, byte-level substring count analysis based on the index's capabilities.
- Retrieve specific documents from the news corpus based on the search API functionality.
- Build or prototype search services without constructing the index from scratch based on the pre-built availability.
Strengths
- Pre-built indexes eliminate the need for users to construct them from scratch.
- Enables exact substring counting and document retrieval as described.
- Powers a live, keyless REST API for search and n-gram counts.
Limitations
- Description metadata is limited; actual data quality requires manual inspection after download.
- Column-level documentation is absent; field semantics must be inferred after download.
- Row count and file formats are unknown, which may limit suitability assessment.
Provenance
- Source
- ruggsea/infini-news-corpus parquets.
- Collection Method
- Built with infini-gram-mini (Burrows–Wheeler Transform + suffix array).
- Time Range
- null
- Freshness
- Last updated 2026-07-01 11:47:36; freshness should be verified.
- Geography
- null