LiveSQLBench: A Dynamic, Contamination-Free Benchmark for Text-to-SQL

LiveSQLBench-Base-Lite is a dynamic benchmark for evaluating large language models on complex, real-world text-to-SQL tasks. It is maintained by the BIRD Team at HKU and Google Cloud. The dataset is described as contamination-free and continuously evolving, with a last recorded update in May 2026.

Use Cases

• Benchmarking LLM performance on text-to-SQL conversion based on its described role as an evaluation benchmark.
• Training or fine-tuning SQL generation models based on the complex, real-world tasks it contains.
• Studying model contamination in NLP benchmarks based on its stated contamination-free design.
• Analyzing the evolution of model capabilities over time based on the benchmark's continuously updated nature.

Strengths

• Designed to be contamination-free, a specific feature for reliable LLM evaluation.
• Continuously evolving, suggesting ongoing updates and relevance.
• Focuses on complex, real-world text-to-SQL tasks, indicating practical application.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.
• Description metadata is limited; actual data quality requires manual inspection after download.

Provenance

Source

BIRD Team at HKU and Google Cloud

Collection Method

Likely curated for benchmarking LLMs on text-to-SQL tasks.

Freshness

Last updated 2026-05-25 12:19:26; freshness should be verified.