Outlier Detection for Distributed Services using Multi-Frequency Patterns

Outlier detection has been commonly used in large scale distributed systems to detect abnormalities and service failures. Traditional abnormality detection methods are mostly based on statistical analysis or simple thresholds to detect outliers. However, those solutions do not consider the periodic patterns of data which is a common feature in many systems due to users' repetitive activities on a daily basis. In our work, we investigate a method that can significantly improve the effectiveness of outliers detection through utilizing periodic patterns in the system data. To this purpose we use State-Frequency-Memory (SFM) Recurrent Neural Networks (RNN) to analyze the cyclical patterns of monitoring data and use cumulative sum (CUMSUM) to identify outliers. Our real data evaluation shows that our detection method can achieve 99% accuracy. Finally, our method also applies to many other fields, such as network traffic security monitoring, bank fraud detection, etc.