A survey of Detecting Flakiness in Automated Test Regression Suite

Recently, Flaky Tests have gotten the attention of the research community. These tests not only waste time and resources, but also undermine the reliability of the automated tests and hence the software systems are affected. So, the detection of Flaky Tests is very important and has a wide audience from both industry and academia. In addition, the research work is very promising having a great potential for the contribution that maps to a true value added to the up-to-date industry projects. Although lots of effort has been done to detect Flakiness in the software systems, the majority of the existed works focuses on different techniques in isolation. The aim of the work in this paper is to overlook the existed flakiness detection techniques, try to analyze them, and hold a comparison, indicating the proper context for each one. Most of the existed flakiness detection techniques are done on unit tests and library/technical projects. According to the comparative study among these techniques, it is found gaps and wide potential for contribution. On the other hand, flakiness detection techniques can be clustered under three main headers; First, study the phenomena under different testing levels by trying to focus on projects with integration, system, and end-toend tests. Second, study the phenomena under different Project types and data sets by studying business projects with user interfaces and mobile applications and so on. Third, study the phenomena toward a faster delivery cycle by studying flake tests in the context of parallel execution and Continuous Integration and Continuous Deployment (CI/CD). We can conclude that not all detection techniques have the same potential where some techniques are very competent while others still struggle to achieve considerable efficiency.