System Logs Anomaly Detection Using Deep Learning

In this paper, we propose a solution based on deep learning methods to face the growth of number of logs in system log data. The technique includes the collection of log data followed by these data passing through LSTM neural nets to be ableto infer temporal items. Through training LSTM over multiple historical samples, we form the rule set capturing that normality of the system. In stark contrast to prior detection methodologies that have been centered exclusively around anomaly types that are known, our model is driven by the goal of detecting unknownanomalies to which no category is assigned in advance. The Hadoop Distributed File System (HDFS log dataset) that hasbeen provided by experts in the domain, is used by us. The approach to be presented offers a competitive way of performing log analysis of systems with a notable capacity of identifying the abnormal activity of the systems of scales in processing operations. Since the development of the project is based on the principles of simplicity and scalability. We enable identification of abnormal patterns with less effort by systematically streamlining this process consequently saving operations time and utilization of labor resources.