Proactive failure detection learning generation patterns of large-scale network logs

With the growth of services in IP networks, network operators are required to perform proactive operation that quickly detects the signs of critical failures and prevents future problems. Network log data, including router syslog, are rich sources for such operations. However, it has become impossible to find genuinely important logs that lead to serious problems due to the large volume and complexity of log data. We propose a log analysis system for proactive detection of failures. Our key observation is that the abnormality of logs depends on not just the keywords in the messages (e.g. ERROR, FAIL), but generation patterns such as burstiness. Our system consists of three functions: (i) extracting log templates automatically and quickly from a massive amount of unstructured log data; (ii) constructing log feature vectors to characterize the generation patterns of logs; and (iii) using a supervised machine learning approach to associate failures with the log data that appeared before them. We validated our system using real log data collected from a large network and determined its effectiveness.