Detecting Incipient Faults in Software Systems: A Compressed Sampling-Based Approach

The volume of data to be collected and processed for effective real-time monitoring of large-scale computing systems and networks poses significant Big Data challenges, and a scalable solution requires a systematic approach to dimensionality reduction during the data collection, transmission, and analysis phases. Compressive sampling can reduce the dimensionality of the data collected at the source prior to transmission to the monitoring station. Exploiting the fact that the compressed samples preserve in approximate form, the correlation information between data points in the original full-length signal, we develop a low-cost anomaly detection technique based on principal component analysis (PCA) aimed at incipient faults such as software aging—the key idea being PCA is performed directly on the compressed samples without having to reconstruct the original signal. Using case studies involving long-running enterprise benchmark applications, Trade6 and RuBBoS, with injected memory leaks, we show that the performance of the PCA-based detector when using just the compressed data is almost equivalent to the case in which the raw data is completely available, but achieved using significantly fewer samples with a compression rate exceeding 75%.