Commit Message Edits by Experts for AI-Generated Text Evaluation

JetBrains-Research provides a dataset of expert-labeled edits to AI-generated commit messages. The data was created by presenting labelers with GPT-4 generated messages for 15 commits from the CMG benchmark and asking them to manually edit the messages to a quality suitable for version control systems. The dataset was last updated on 2024-10-17.

Use Cases

• Training or fine-tuning models for commit message generation based on expert-corrected examples.
• Benchmarking the quality of AI-generated commit messages against human-edited standards.
• Studying the types of edits experts make to improve AI-generated text in a software engineering context.

Strengths

• Data is expert-labeled, suggesting a high-quality reference standard.
• Based on a defined benchmark (CMG from Long Code Arena) for controlled evaluation.
• Last update timestamp is explicitly provided (2024-10-17 20:22:53).

Limitations

• Row count is unknown, which may limit suitability assessment.
• Column-level documentation is absent; field semantics must be inferred after download.
• The dataset is based on only 15 commits, which may limit statistical representativeness.

Provenance

Source

JetBrains-Research via Hugging Face.

Collection Method

Labelers manually edited GPT-4 generated commit messages via a dedicated app, as described in the paper 'Towards Realistic Evaluation of Commit Message Generation by Matching Online and Offline Settings'.

Time Range

null

Freshness

Last updated 2024-10 17 20:22:53; freshness should be verified.

Geography

null