MT-Consistency: A Benchmark for Multi-Turn LLM Consistency Evaluation

MT-Consistency is a curated benchmark for evaluating how well large language models maintain correct judgments over multi-turn interactions. The benchmark combines controlled multiple-choice questions with eight types of follow-up prompts that probe susceptibility to pressure, tone, and misinformation. It was created by author yubol and was last updated on the platform on 2025-09-21.

Use Cases

• Evaluate LLM consistency under pressure based on follow-up prompts designed to apply pressure.
• Assess LLM susceptibility to misinformation based on follow-up prompts that introduce false information.
• Test mitigation methods for improving model consistency based on the benchmark's evaluation protocols.
• Compute Position-Weighted Consistency (PWC) metrics based on the benchmark's defined evaluation framework.

Strengths

• Benchmark includes eight distinct types of follow-up prompts designed to probe specific weaknesses.
• Supports two distinct evaluation protocols: Repetitive and Diverse follow-ups.
• Includes defined metrics such as Position-Weighted Consistency (PWC) for structured evaluation.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment for large-scale training.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

yubol on Hugging Face

Collection Method

Curated benchmark, likely constructed for evaluation purposes.

Freshness

Last updated 2025-09-21 14:32:12; freshness should be verified.