Efficient Learning Framework for Failure Identification Model Based on Failure Injection

Failure injection is an effective approach to accumulate prior knowledge in failure handling. In recent years, approaches that build machine learning (ML) models for failure handling on the basis of data generated by failure injection have attracted attention. In general, however, the space of failures/parameters to be explored is huge and the failure injection cycle involves very time- and manpower-intensive tasks, so an efficient failure injection method is in great demand. In this paper, we propose a failure injection framework for efficiently training a ML-based failure identification model (FIM). The key to our framework is the introduction of another ML model for estimating failure data, called an observation estimator. By using this estimator to estimate the system observations for failures/parameters, whether or not the failure/parameter is effective in training the FIM can be judged in advance before it is actually injected into the system. Only failures/parameters that are determined to be highly effective are preferentially injected into the system, thereby obtaining informative training data that speeds up the learning of the FIM. We applied this framework to an e-commerce site built with Kubernetes and evaluated it. Results demonstrated that the framework enables the FIM to quickly reach the target accuracy with fewer failure injections than the baselines.