sysevol-ai created a collection of natural-language code-search evaluation queries synthesized by LLMs. Each query is grounded on a real code symbol from a SWE-bench instance. The dataset is in active development, with counts representing a current snapshot.
Use Cases
- Benchmarking code search models based on LLM-synthesized queries
- Training retrieval models for software engineering tasks based on grounded code symbols
- Evaluating the performance of natural language to code translation systems
Strengths
- Queries are grounded on real code symbols from SWE-bench instances
- Dataset is actively growing, indicating ongoing curation
Limitations
- Description metadata is limited; actual data quality requires manual inspection after download
- Column-level documentation is absent; field semantics must be inferred after download
- Row count is unknown, which may limit suitability assessment
Provenance
- Source
- sysevol-ai
- Collection Method
- LLM-synthesized queries grounded on SWE-bench code symbols
- Freshness
- Last updated 2026-05-21 18:32:21; freshness should be verified