Large-scale Log-based Failure Diagnosis of Server Groups: A Two-stage Mining Approach Based on Drain 3 and Weight-based Optimization Algorithm

In recent years, cloud computing centers have developed rapidly due to the increasing number of big data services. However, large-scale server groups are also accompanied by a high incidence of faults, so it is unrealistic to rely on manual diagnosis. Therefore, the researchers try to mine the log files with machine learning algorithms to achieve automatic failure diagnosis. However, the unstructured properties of log data and the constraints that machine learning requires large training sets have become the stumbling blocks on the research road. In order to improve the effect of fault diagnosis, based on previous studies, this paper proposes a two-stage method: In the first stage, parsing logs by Drain3 which is used to extract and match the template of log messages. In the second stage, the model is trained and outputs predictions which are mixed based on the Weight-based optimization algorithm based on RandomForest and CatBoost. Thus, the prediction results are optimized. After comparative tests, the proposed approach outperformed the other nine approaches. Moreover, due to the synthesis of Bagging and Boosting algorithms, the robustness of this approach is also improved.