A Flexible and Efficient Alert Correlation Platform for Distributed IDS

Intrusion Detection Systems (IDS) have been widely deployed in practice for detecting malicious behavior on network communication and hosts. The problem of false-positive alerts is a popular existing problem for most of IDS approaches. The solution to address this problem is correlation and clustering of alerts. To meet the practical requirements, this process needs to be finished as soon as possible, which is a challenging task as the amount of alerts produced in large scale deployments of distributed IDS is significantly high. We identify the data storage and processing algorithms to be the most important factors influencing the performance of clustering and correlation. We propose and implement the utilization of memory-supported algorithms and a column-oriented database for correlation and clustering in an extensible IDS correlation platform. The utilization of the column-oriented database, an In-Memory Alert Storage, and memory-based index tables leads to significant improvements on the performance. Different types of correlation modules can be integrated and compared on this platform. A plugin concept for Receivers provides flexible integration of various sensors and additional IDS management systems. The platform can be distributed over multiple processing units to share memory and processing power. A standardized interface is designed to provide a unified view of result reports for end users. The efficiency of the proposed platform is tested by practical experiments with several alert storage approaches, different simple algorithms, as well as local and distributed deployment.