QAngaroo: Multi-hop Reading Comprehension Datasets Across Documents

Two datasets from different domains designed for multi-hop reading comprehension, where models must combine evidence from multiple documents. The best model accuracy on an annotated test set is 54.5%, compared to human performance at 85.0%. The dataset was created by Johannes Welbl of University College London to investigate the limits of existing text understanding methods.

Use Cases

• Benchmarking multi-hop reading comprehension models based on the task of seeking and combining textual evidence.
• Training models for text understanding across multiple documents based on the described query-answer pairs and thematically linked documents.
• Investigating model limitations in selecting relevant information, as the description notes performance improves when provided guaranteed relevant documents.

Strengths

• Human performance benchmark of 85.0% provides a clear target for model improvement.
• Two datasets from different domains offer varied thematic contexts for evaluation.
• Methodology includes identified pitfalls and circumvention strategies for dataset construction.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Last update date is unknown; freshness unverified.

Provenance

Source

University College London

Collection Method

Induced via a methodology using query-answer pairs and thematically linked documents.