AI Driven Methodology for Anomaly Detection in Apache Spark Streaming Systems

Cloud computing, Artificial Intelligence, and Big Data technologies have recently become one of the most impactful forms of technology innovation. It is common to have multiple users share the same computing resources. This practice noticeably leads to performance anomalies. For instance, some applications can feature variability in processing time due to interference from other applications, or software contention from the other users, which may lead to unexpectedly long execution time and be considered anomalous. There is an urgent need for an automated effective performance anomaly detection method that can be used within the production environment for the streaming system to avoid any late detection of unexpected system failures. To address this challenge, we introduce a new black-box training workload configuration optimization with a neural network driven methodology to identify anomalous performance in an in-memory Spark streaming Big Data platform. The proposed methodology effectively uses Bayesian Optimization to find the ideal training dataset size and Spark streaming workload configuration parameters to train the anomaly detection model. The proposed model is validated on the Apache Spark streaming system. The results demonstrate that the proposed solution succeeds and accurately detects many types of performance anomalies. In addition, the training time for the machine learning model is reduced by more than 50%, which offers a fast anomaly detection deployment for system developers to utilize more efficient monitoring solutions.