Anomaly Detection on Real-time Security Log using Stream Processing

Many critical tasks such as document approval and banking services, which are now hosted on cloud infrastructure. This transformation introduces stress on cloud security from the physical layer of the data center to the application layer of web application. All data access and service access need to be monitored and responded to in real-time. In this paper, we study methods to detect anomaly incidents such as spikes from network volume, malicious incidents from API scanning, error messages from internal systems and timeout from Slowloris attack[l]. We select machine learning based anomaly detection algorithms, such as LOF, Isolation Forest and Elliptic Envelope, to find suitable methods to detect incidents in real-time using stream processing tools including Kafka and message ingression. The result shows that LOF is fast and robust in most of the cases. However, when log messages have unseen words, which normally need to be hashed to preprocess, the Isolation Forest shows better results. This study shows the possibility of applying stream processing with machine learning to detect anomaly behavior for cloud services.