A Deep Learning Approach for Real-Time Application-Level Anomaly Detection in IoT Data Streaming

The growth of streaming data originating from Internet of Things (IoT)-based Industry 4.0 opens doors to real-time analytics of time-sensitive services. However, this ever-increasing amount of data inevitably leads to anomalies, resulting in considerable risks for time-sensitive applications. Thus, real-time detection of anomalies is critical to prevent impending failures and resolve them in time. Given that the problem is to detect application-level anomalies in real time, we develop a deep learning-based technique, which integrates time-series data inference with a Long-Short Term Memory (LSTM)-based prediction model. Our proposed method relies on a novel metric called Sequence Inconsistency Distance (SID), which determines the abnormality likelihood of a target record in real time. Our trace-driven evaluations indicate that the proposed method achieves up to a 92.6% performance gain compared to the current state-of-the-art anomaly detection methods in terms of true positive and false positive rate while meeting the essential efficiency requirements.