LiveBench Coding: 1,000 Contamination-Resistant LLM Benchmarks

LiveBench Coding is a benchmark for Large Language Models containing approximately 1,000 coding-related questions designed to prevent test set contamination. Created by the LiveBench team and updated as recently as April 2025, the dataset utilizes objective ground-truth answers derived from recent sources like arXiv papers and news articles. It is released on a monthly cycle to ensure that the evaluation material remains outside the training window of most models.

Use Cases

• Evaluating LLM code generation accuracy against objective ground-truth answers
• Benchmarking model performance on recent coding paradigms not present in older training sets
• Measuring contamination levels in LLMs by comparing performance on monthly released subsets

Strengths

• Monthly release cycle specifically designed to mitigate data contamination
• Objective ground-truth answers for verifiable and automated scoring
• Linked to Arxiv 2406.19314 for methodology transparency

Limitations

• Small sample size of approximately 1,000 records
• Specific focus on coding tasks may not reflect general-purpose reasoning capabilities

Provenance

Source

LiveBench (Arxiv 2406.19314)

Collection Method

Curated from recently-released datasets, arXiv papers, news articles, and IMDb synopses

Time Range

2024-2025

Freshness

Updated monthly to include new questions and mitigate data leakage.

Geography

Global