A Note on Optimal Retraining Strategy for ML Systems

In this paper, we discuss the maintenance problem of ML (Machine Learning) systems. In general, the ML system is fine-tuned with the data observed from the operational environment before deployment. However, it is known that the performance of the ML system degrades with time. This is called model drift. There are a variety of causes of model drift, such as the degradation of the hardware machine itself and the climate condition. To deal with the model drift, ML system should be maintained with retraining. The retraining is to perform fine tuning of ML systems with the data collected from the current operating environment. On the other hand, the retraining of ML system causes the maintenance cost, so it is important to make a balance between the retraining cost and the penalty of degraded system. This paper presents probabilistic models to represent the retraining strategy for a simple ML system with discriminator. Specifically, we consider the periodic retraining strategy and the detector-based retraining strategy in our probabilistic models. Through the numerical experiment, these strategies are compared in terms of maintenance cost.