Loading [MathJax]/extensions/MathMenu.js
LogTransfer: Cross-System Log Anomaly Detection for Software Systems with Transfer Learning | IEEE Conference Publication | IEEE Xplore

LogTransfer: Cross-System Log Anomaly Detection for Software Systems with Transfer Learning


Abstract:

System logs, which describe a variety of events of software systems, are becoming increasingly popular for anomaly detection. However, for a large software system, curren...Show More

Abstract:

System logs, which describe a variety of events of software systems, are becoming increasingly popular for anomaly detection. However, for a large software system, current unsupervised learning-based methods are suffering from low accuracy due to the high diversity of logs, while the supervised learning methods are nearly infeasible to be used in practice because it is time-consuming and labor-intensive to obtain sufficient labels for different types of software systems. In this paper, we propose a novel framework, LogTransfer, which applies transfer learning to transfer the anomalous knowledge of one type of software system (source system) to another (target system). We represent every template using Glove, which considers both global word co-occurrence and local context information, to address the challenge that different types of software systems are different in log syntax while the semantics of logs should be reserved. We apply an LSTM network to extract the sequential patterns of logs, and propose a novel transfer learning method sharing fully connected networks between source and target systems, to minimize the impact of noises in anomalous log sequences. Extensive experiments have been performed on switch logs of different vendors collected from a top global cloud service provider. LogTransfer achieves an averaged 0.84 F1-score and outperforms the state-of-the-art supervised and unsupervised log-based anomaly detection methods, which are consistent with the experiments conducted on the public HDFS and Hadoop application datasets.
Date of Conference: 12-15 October 2020
Date Added to IEEE Xplore: 11 November 2020
ISBN Information:

ISSN Information:

Conference Location: Coimbra, Portugal

Contact IEEE to Subscribe

References

References is not available for this document.