Manually Curated Adversarial Question Answering Examples

Comprising manually curated adversarial examples designed to be more difficult than the original SQuAD adversarial dataset. It was created by author shivmoha and last updated in December 2021. Specific details on row count, column structure, and file formats are unavailable.

Use Cases

• Evaluate the robustness of question answering models against manually crafted adversarial examples.
• Benchmark model performance on a curated set of challenging question-answer pairs.
• Analyze failure modes of NLP systems when presented with difficult, manually created adversarial inputs.

Strengths

• Examples are manually curated by the authors, suggesting intentional design for difficulty.
• Dataset is explicitly designed to be more challenging than the original SQuAD adversarial dataset.
• Last update recorded as December 17, 2021, providing a timestamp for the curation effort.

Limitations

• The dataset size, structure (columns), and specific content are unknown, limiting reproducibility and analysis.
• The dataset's age may not reflect the latest state-of-the-art model capabilities or adversarial techniques.
• Without sample data or a detailed schema, the exact nature of the adversarial modifications is unclear.

Provenance

Source

huggingface, author shivmoha

Collection Method

Manually curated by the authors based on the concept of the SQuAD adversarial dataset.

Freshness

Last updated 2021-12-17.