Credibility-Aware Generation Benchmark for Question Answering

Credibility-aware Generation Benchmark (CAGB) evaluates model performance in handling flawed information across three scenarios: Open-domain QA, Time-sensitive QA, and Misinformation Polluted QA. The benchmark includes datasets like 2WikiMultiHopQA, HotpotQA, Musique, RGB, EvoTempQA, and NewsPollutedQA. It was created by ruotong-pan and updated in April 2024.

Use Cases

• Evaluate model credibility-aware generation ability on the Open-domain QA scenario using datasets like 2WikiMultiHopQA and HotpotQA.
• Assess model performance on the Time-sensitive QA scenario with the EvoTempQA dataset to handle temporal information.
• Test models on the Misinformation Polluted QA scenario using the NewsPollutedQA dataset to identify and correct flawed information.

Strengths

• Benchmark covers three distinct scenarios essential for evaluating credibility-aware generation.
• Includes multiple established QA datasets such as HotpotQA and NewsPollutedQA.
• Dataset was updated in April 2024, indicating recent maintenance.

Limitations

• Specific dataset size, row count, and column structure are unknown.
• Limited to English language data, restricting multilingual evaluation.
• The benchmark's scope is defined by its three scenarios, potentially excluding other credibility contexts.

Provenance

Source

ruotong-pan via Hugging Face.

Collection Method

Constructed as a benchmark from existing QA datasets for evaluating model credibility.

Freshness

Last updated on 2024-04-11.