EVOSUITE Test Smells: A Benchmark for Detection Tools

2,340 test cases generated by EVOSUITE for 100 Java classes form a manually labeled dataset for six types of test smells. This dataset was created to benchmark the performance of test smell detection tools on automatically generated test suites. The analysis reveals that existing detection strategies misclassified over 70% of test smells in this context, highlighting patterns for tool improvement.

Use Cases

• Benchmarking test smell detection tools based on manually validated labels.
• Studying the characteristics of test smells in automatically generated Java test code.
• Improving the definitions and detection rules for test smells based on identified patterns.
• Training or evaluating machine learning models for software test quality assessment.

Strengths

• Contains 2,340 manually analyzed and labeled test cases, providing a ground truth for evaluation.
• Focuses on a specific and relevant context: test smells in EVOSUITE-generated tests for 100 Java classes.
• Labels were derived from a multi-stage, cross-validated manual analysis, suggesting methodological rigor.

Limitations

• The specific column names and data schema are not provided in the source descriptions.
• Key metadata such as the exact creation date, last update, and file formats are unknown.
• The dataset size is fixed to the described 2,340 cases; its scale for broader generalization is limited.

Provenance

Source

Annibale Panichella, Delft University of Technology

Collection Method

Test cases were automatically generated by EVOSUITE, followed by multi-stage, cross-validated manual analysis for labeling.