CodeReview-Bench: 100K+ GitHub Code Editing and Review Pairs

CodeReview-Bench is a software engineering benchmark curated by ronantakizawa for evaluating models on code editing and review tasks. It contains between 100,000 and 1,000,000 records derived from GitHub interactions, updated as of March 2026. The dataset is structured to support sequence-to-sequence tasks where natural language feedback is converted into code modifications.

Use Cases

• Generating 'after_code' from 'before_code' and 'reviewer_comment' inputs.
• Analyzing model performance across different programming 'language' categories.
• Evaluating the utility of 'diff_context' in predicting precise code modifications.

Strengths

• 100,000 to 1,000,000 records
• MIT license
• Includes 'diff_context' for localized code changes

Limitations

• Potential for label noise due to the organic nature of GitHub comments
• Geographic bias toward US-based data as indicated by dataset tags
• Lack of manual verification for the correctness of the 'after_code' targets

Provenance

Source

ronantakizawa/github-codereview

Collection Method

curated from GitHub

Freshness

Updated March 2026.

Geography

US