Anomaly Detection Mechanism Based on Hierarchical Weights through Large-Scale Log Data

In order to realize Intelligent Disaster Recovery and break the traditional reactive backup mode, it is necessary to forecast the potential system anomalies, and proactively backup the real-time datas and configurations. System logs record the running status as well as the critical events (including errors and warnings), which can help to detect system performance, debug system faults and analyze the causes of anomalies. What's more, with the features of real-time, hierarchies and easy-access, log data can be an ideal source for monitoring system status. To reduce the complexity and improve the robustness and practicability of existing log-based anomaly detection methods, we propose a new anomaly detection mechanism based on hierarchical weights, which can deal with unstable log data. We firstly extract semantic information of log strings, and get the word-level weights by SIF algorithm to embed log strings into vectors, which are then feed into attention-based Long Short-Term Memory(LSTM) deep learning network model. In addition to get sentence-level weight which can be used to explore the interdependence between different log sequences and improve the accuracy, we utilize attention weights to help with building workflow to diagnose the abnormal points in the execution of a specific task. Our experimental results show that the hierarchical weights mechanism can effectively improve accuracy of perdition task and reduce complexity of the model, which provides the feasibility foundation support for Intelligent Disaster Recovery.