FlakeRake: Results for Reproducing Timing-Dependent Flaky Tests

FlakeRake is a tool for automatically reproducing timing-dependent flaky-test failures. This artifact contains the source code for FlakeRake, raw and processed results from its evaluation, and failure messages extracted from the FlakeFlagger dataset. The data was produced by Shanto Rahman at The University of Texas at Austin.

Use Cases

• Evaluating the effectiveness of automated flaky-test reproduction tools based on the provided failure messages and rerun results.
• Analyzing patterns in timing-dependent API calls that cause test failures based on the included analysis CSV file.
• Comparing different execution modes (bisection vs. one-by-one) for reproducing flaky tests using the provided results archives.
• Replicating the FlakeFlagger study using the provided raw rerun results and execution scripts.

Strengths

• Includes raw and processed results from a large-scale evaluation, such as failure messages from the FlakeFlagger dataset.
• Contains analysis of timing-dependent API calls linked to specific failure configurations.
• Provides execution scripts for running experiments on an HPC cluster, aiding reproducibility.

Limitations

• Row count and dataset size are unknown, which may limit suitability assessment.
• Column-level documentation is absent; field semantics must be inferred after download.
• The FlakeRake source code is anonymized and not runnable in its current state.

Provenance

Source

Shanto Rahman, The University of Texas at Austin

Collection Method

Artifact from the evaluation of the FlakeRake tool, including results from rerunning tests from the FlakeFlagger dataset.