FALL: Prior Failure Detection in Large Scale System Based on Language Model

As the scale of high-performance computing systems (HPC) continues to expand, the frequency of failures within the system also increases. The follow-up steps in the event of a failure are necessary because efficient prevention measures are absent. These actions, however, result in decreased operating efficiency as the system normalizes. To solve this problem, recent research has utilized deep learning to predict failures in advance by examining log messages. One common method uses log ID extracted from log messages as input data. However, these methods could lead to information loss in log messages and may not accurately capture log message properties. In this study, we propose the prior failure detection in large-scale systems using language models (FALL) that reflects the properties of log messages. The FALL uses log messages as text data and only uses normal data for training. The FALL makes use of sharpening, which concentrates on the minimal lexical change between normal and abnormal log messages. In addition, by leveraging the characteristics of log messages with limited vocabularies, the FALL utilizes only some tokens for anomaly detection. The experiments’ results from log data from real industry-based HPC systems show that the FALL achieves superior performance in early failure detection.