DeepEAD: Explainable Anomaly Detection from System Logs

System logs record rich information for system events. Practical anomaly detection from system logs should be able to address three challenges: 1) understanding complicated attributes in event logs; 2) extracting complex context relations among events; and 3) providing concrete explanations to human analysts. In this paper, we develop an attention-equipped encoder-decoder system to capture context from system logs for explainable anomaly detection. For each target event, we collect its nearby events in chronological order as its context events. Instead of using a recurrent neural network-based encoder like previous works, we adopt a Transformer-based encoder to extract complex relations among context events and their attributes. Then, a context vector is generated and passed to the decoder, where an attention matrix is learned and used to weigh the context events for detecting the anomalies. Evaluation on the large-scale real-world Los Alamos National Laboratory dataset shows that, compared with existing works, our methods can provide fine-grained one-to-one attention to help explain the importance of each attribute in the context events to the prediction, without sacrificing detection performance.

I. Introduction

Available in almost all computer systems, logs are used to record various events for monitoring, administration, and debugging, which provide a good source of information for analyzing and identifying anomalies. Since modern IT infrastructure systems continuously generate an overwhelming amount of event logs and attacks are evolving and becoming more complex [1], automated anomaly detectors are usually applied to flag potential anomalies. Then, detected events will be handed over to human analysts for further analysis [2]. However, as reported in FireEye M-Trends 2021 [3], the median time for organizations to identify incidents by the help of anomaly detectors is 24 days, yielding too much time to attackers for conducting malicious activities. This is due to two major weaknesses of existing anomaly detectors: high false-positive rate and lack of explanations in detection results.