Online anomaly detection in big data

In this paper, the problem of online anomaly detection in multi-attributed, asynchronous data from a large number of individual devices is considered. It has become increasingly common for many services, such as video-on-demand (VOD), to have connected customers where hundreds of millions of subscribers access a cluster of content servers for online services. It is important to monitor these transactions online, in order to ensure acceptable quality of experience to the customers as well as for detecting any abnormal or undesirable activities. Our proposed anomaly detection strategy works in two phases: First we perform intermittent anomaly detection in space, using data from the entire set of devices for a short duration in time. This phase employs principal component analysis (PCA) for data reduction and captures models of normal and abnormal features. Then, these identified models are used to monitor each subscriber's devices online in order to quickly detect any abnormalities. The proposed approach is demonstrated on Comcast's Xfinity video streaming data.