Log-TF-IDF for Anomaly Detection in Network Switches

In this study, we focused on anomaly detection of network switches using log analysis. In previous research, we designed a log parser to find common patterns within logs, which replaced logs with ’pattern’ and classified similar patterns into the same ’event’. In this study, we utilize the log parser and calculate abnormality scores for each log pattern. In log analysis, analyzing the occurrence patterns is more important than the meaning of individual logs. With this perspective, we design the following three items to analyze the tendency of occurrence of each log pattern to better suit network switch log analysis. 1) Log-TF-IDF: the computed rarity of each log pattern, 2) Log-Prob: the probability of occurrence of each event predicted by machine learning, 3) Log-Freq: the numerical value for the degree of repetition of each log event. We calculate the ’Abnormal Score’ by considering all three metrics. By calculating Abnormal Scores from both normal and abnormal log data, we demonstrate the effectiveness of our approach. We implemented a simple anomaly detection system utilizing our proposed Abnormal Scores and achieved an F1 score of 88.2% in detecting anomalies within the log data collected from L2 and L3 switches.