ISO-Bench: 54 Performance Optimization Commits from vLLM and SGLang

ISO-Bench consists of 54 human-authored performance optimization commits sourced from the vLLM and SGLang repositories, updated in February 2026. Created by Lossfunk, this dataset provides original code diffs and benchmark commands to evaluate AI agents on software performance tasks.

Use Cases

• Evaluating AI agents on code optimization by comparing the original diff with generated improvements
• Training models to interpret 'perf_command' outputs for performance regression testing
• Fine-tuning LLMs for software engineering tasks using commit messages as natural language labels

Strengths

• 54 real-world optimization commits from production-grade LLM engines
• Includes executable 'perf_command' for verification
• Human-authored data from expert developers

Limitations

• Small sample size of 54 records
• Narrow domain focus on LLM inference and serving frameworks

Provenance

Source

vLLM and SGLang GitHub repositories

Collection Method

Curated from open-source software performance commits

Freshness

Last updated February 2026.