EvalEng: GitHub Issues from 57 ML Evaluation Harnesses

EvalEng is a dataset of GitHub issues collected from 57 machine learning evaluation harnesses. The issues are annotated with workflow stage, root cause, and evaluation strategy labels. The dataset was created by author zhiminy and is associated with the paper 'Towards Evaluation Engineering'.

Use Cases

• Analyzing common failure modes in ML evaluation pipelines based on annotated root cause labels
• Studying the software engineering lifecycle of evaluation tools based on annotated workflow stage labels
• Categorizing evaluation strategies and methodologies used across different ML projects
• Training models to automatically triage or summarize issues for evaluation harnesses

Strengths

• Issues sourced from 57 distinct ML evaluation harnesses
• Issues are annotated with three specific label types: workflow stage, root cause, and evaluation strategy

Limitations

• Column-level documentation is absent; field semantics must be inferred after download
• Row count is unknown, which may limit suitability assessment

Provenance

Source

GitHub repositories of 57 ML evaluation harnesses

Collection Method

Issues collected and annotated as part of the EvalEng research project

Freshness

Last updated 2026-05-27 04:18:38; freshness should be verified