Long short-term memory based operation log anomaly detection

Long short-term memory (LSTM) architecture is an important approach for capturing long-range temporal dependencies in sequences of arbitrary length. Moreover, stacked-LSTM (S-LSTM: formed by adding recurrent LSTM layer to the existing LSTM network in hidden layer) has capability to learn temporal behaviors quickly with sparse representations. To apply this to anomaly detection, we model the operation log samples of normal and anomalous events occurred in 1 minute time interval as time-series with the aim to detect and classify the events as either normal or anomalous. To select an appropriate LSTM network, experiments are conducted for various network parameters and network structures with the dataset provided by Cyber Security Data Mining Competition (CDMC2016). The experiments are run up to 1000 epochs with learning rate in the range [0.01-05]. S-LSTM network architecture has showed its strength by achieving the highest accuracy 0.996 with false positive rate 0.02 on the provided real-world test data by CDMC2016.