GitHubr Major Repos: Metadata-Only Commit History for Large-Scale Code Models

A metadata-only dataset of GitHub commits designed for large-scale AI and software engineering research, created by adhyanshaa and last updated on June 5, 2026. It aims to solve the problem of training models on open-source software history without managing large volumes of raw code. The dataset is accompanied by the GitScope CLI tool for exploration.

Use Cases

• Pretraining large language models for code based on commit metadata sequences.
• Analyzing software development workflows and patterns from commit history.
• Studying the evolution of major open-source repositories over time.
• Training models for tasks like commit message generation or bug prediction.

Strengths

• Designed specifically for large-scale model pretraining, as stated in the description.
• Focuses on metadata to reduce storage and management overhead for researchers.
• Includes a companion CLI tool (GitScope) for dataset exploration.

Limitations

• Description metadata is limited; actual data quality requires manual inspection after download.
• Column-level documentation is absent; field semantics must be inferred after download.
• Row count, file formats, and license information are unknown.

Provenance

Source

GitHub, likely from major repositories.

Collection Method

Likely gathered via API or Git history parsing, focusing on commit metadata.

Time Range

null

Freshness

Last updated 2026-06-05 13:03:48; freshness should be verified.

Geography

null