Flaky Tests in the AI Domain

In this position paper, we investigate how frequently is randomness the cause of flakiness in the traditional and in the AI-enabled software domains. Based on previous works, it seems that while in the general domain flakiness rarely stems from randomness, in the AI domain it is a frequent phenomenon. Thus, we urge a discussion about a classification scheme of flaky tests based on whether they are caused by the inherent randomness of the AI-enabled SUT or some other reason. This way, better identification, classification and proper handling of flakiness in such systems will be possible.CCS CONCEPTS• Software and its engineering → Software testing and debugging; • Theory of computation → Pseudorandomness and derandomization; • Computing methodologies → Artificial intelligence; Machine learning.