GitGoodBench Lite is a 120-sample subset for evaluating AI agent performance on Git tasks. The dataset was created by JetBrains and last updated on 2025-11-18. It contains an even split of samples across Python, Java, and Kotlin programming languages and the task types of merge conflict resolution and file-commit gram.
Use Cases
- Benchmarking AI agent performance on Git merge conflict resolution based on the described sample type.
- Evaluating AI agent capabilities for file-commit operations based on the described sample type.
- Comparing AI agent effectiveness across different programming languages (Python, Java, Kotlin) based on the dataset's language split.
- Training or fine-tuning models for automated software engineering tasks based on the described Git scenarios.
Strengths
- Contains 120 samples, providing a defined evaluation corpus.
- Structured with 20 samples per combination of the two task types and three programming languages.
- Data is sourced from 100 unique, open-source GitHub repositories, suggesting real-world provenance.
Limitations
- Column-level documentation is absent; field semantics must be inferred after download.
- Row count is limited to 120 samples, which may restrict statistical power for some analyses.
- Description metadata is limited; actual data quality requires manual inspection after download.
Provenance
- Source
- JetBrains
- Collection Method
- Collected from 100 unique, open-source GitHub repositories.
- Time Range
- null
- Freshness
- Last updated 2025-11-18 09:06:11; freshness should be verified.
- Geography
- null