TACO-Benchmark: Text-to-SQL Evaluation for Ambiguous and Cross-Database Queries

TACO (Text-to-SQL with Ambiguous and Cross-database Open-domain queries) is a benchmark for evaluating Text-to-SQL systems on real-world data-lake scenarios. It was created by Akanezora and was accepted for publication at VLDB 2026. The benchmark focuses on challenges absent in prior benchmarks like Spider or BIRD.

Use Cases

• Benchmarking Text-to-SQL model performance based on ambiguous natural language queries mentioned in the description
• Evaluating SQL generation systems on cross-database queries as described in the overview
• Testing the robustness of data-lake query systems against real-world open-data challenges outlined in the benchmark

Strengths

• Designed to address three specific challenges common in real open-data scenarios, differentiating it from benchmarks like Spider
• Accepted for publication at a major database conference (VLDB 2026), suggesting peer-reviewed quality

Limitations

• Column-level documentation is absent; field semantics must be inferred after download
• Row count is unknown, which may limit suitability assessment

Provenance

Source

huggingface

Freshness

Last updated 2026-06-14 11:22:49; freshness should be verified