SecRepoBench: 318 Secure Code Completion Tasks from 27 C/C++ Repositories

318 code completion tasks obtained from 27 popular GitHub C/C++ repositories covering 15 Common Weakness Enumerations (CWEs). The benchmark, created by ai-sec-lab and last updated in November 2025, is built upon the ARVO dataset and is designed to evaluate large language models and agent frameworks for secure code generation.

Use Cases

• Benchmarking standalone LLMs with a context retriever for secure code completion based on repository-level tasks.
• Evaluating agent frameworks with access to an entire repository for secure code generation.
• Assessing model performance against specific software vulnerabilities based on the 15 covered CWEs.
• Comparing different code generation paradigms for security using the provided C/C++ tasks.

Strengths

• Contains 318 specific tasks, providing a concrete scale for evaluation.
• Sourced from 27 popular real-world GitHub repositories, suggesting practical relevance.
• Covers 15 distinct CWEs, indicating a focus on multiple vulnerability types.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment for large-scale training.
• Description metadata is limited; actual data quality and structure require manual inspection after download.

Provenance

Source

Built upon the ARVO dataset, sourced from 27 GitHub C/C++ repositories.

Collection Method

Likely involves extracting code completion tasks and labeling them with CWE information.

Time Range

null

Freshness

Last updated 2025-11-20 01:45:05.

Geography

null