Benchmark for Evaluating Search-Based Crash Reproduction

122 hard-to-reproduce crashes from six open-source projects form a benchmark for evaluating seeding techniques in search-based crash reproduction. The dataset supports the evaluation of behavioral model seeding, a method that learns class usages from the system under test and existing test cases to guide test generation. Results from this benchmark indicate the proposed method outperforms state-of-the-art test seeding and no-seeding approaches by a minimum of 6%.

Use Cases

• Evaluating search-based crash reproduction techniques based on a defined set of crash cases.
• Comparing the effectiveness of different seeding strategies (e.g., behavioral model, test, none) for object creation.
• Training or validating machine learning models for software test generation using real crash stack traces.
• Studying the characteristics of hard-to-reproduce crashes in open-source software projects.

Strengths

• Contains a specific benchmark of 122 hard-to-reproduce crashes, providing a concrete evaluation scale.
• Derived from six real-world open-source projects, ensuring relevance to practical software development.
• Serves as the evaluation basis for a peer-reviewed research method published with Open Access licensing.

Limitations

• The specific data columns, row counts, and file formats are not described in the provided metadata.
• The dataset's creation date and last update timestamp are unknown, limiting assessment of freshness.
• The description is repeated verbatim across multiple platform entries, suggesting sparse, non-detailed metadata.

Provenance

Source

Pouria Derakhshanfar, Delft University of Technology

Collection Method

Likely extracted crash stack traces and related project code from six specified open-source projects.