ORCAS-I: 18 Million Search Query-URL Pairs Annotated with User Intent

ORCAS-I is an annotated version of the ORCAS dataset, containing 18 million unique query-URL pairs labeled with user intents using weak supervision. The dataset was created by Kusa, Wojciech at TU Wien and includes a manually annotated gold test set of 1,000 queries. It provides a taxonomy of user intents based on Broder's classification, refined with subcategories.

Use Cases

• Training intent classifiers based on the three top-level categories (informational, navigational, transactional) in the 'level_1' column.
• Developing fine-grained intent models using the full taxonomy (including factual and instrumental subcategories) in the 'label' column.
• Evaluating model performance on a manually annotated test set ('ORCAS-I-gold.tsv') to benchmark against human judgment.
• Analyzing the relationship between query text, clicked URLs, and user intent for search engine optimization or user behavior research.

Strengths

• Large scale with 18,823,602 query-URL pairs and over 10 million unique queries.
• Includes a manually annotated gold-standard test set of 1,000 queries for reliable evaluation.
• Provides a hierarchical intent taxonomy with two levels of annotation (level_1 and level_2/label).
• Offers a curated 2-million pair subset (ORCAS-I-2M.tsv) for experimental use.

Limitations

• Description metadata is limited; actual data quality requires manual inspection after download.
• Last update date is unknown; freshness unverified.
• Data may reflect temporal and source bias inherent to the original ORCAS dataset from paperswithcode.

Provenance

Source

Annotated version of the ORCAS dataset (Craswell et al., 2020).

Collection Method

Intent labels applied using weak supervision, with a manually annotated test set.