Failure Prediction using Real-Time Error Logs, N-Gram Analysis and Long-Short Term Memory

Traditional system management technologies use log data to determine the factors causing system failures after the fact. However, in system critical information technology services such as banking systems, predicting failures and unexpected system downtime can prevent serious financial losses and ensures reliability. In this study, a new fault prediction method is presented, where N-gram analysis and Long-Short Term Memory (LSTM) are used in the detection of anomalies in bank log data for predicting failures prior to their occurrences. A staged clustering algorithm using N-gram analysis is used in the preprocessing of bank log data for predicting parameter-independent key messages. Instead of using the time-window method, fault prediction is based on sorted log arrays of the requester ID. It is shown that the presented method is able to more effectively determine the fault, compared to the time-window method via experimental work using real-time bank log data. It is also shown that, compared to the literature, the presented method creates a lower workload on the system, since it only uses error logs, and does not additionally require the usage of information logs.