RANK: AI-Assisted End-to-End Architecture for Detecting Persistent Attacks in Enterprise Networks

Modern government and enterprise networks are the target of sophisticated multi-step attacks called Advanced Persistent Threats (APTs), designed and carried out by expert adversaries. The prolonged nature of APTs results in overwhelming the analyst with an increasingly impractical number of alerts. As a result, the challenge of APT detection is ideal for automation through artificial intelligence (AI). In this paper, we propose the first, up to our knowledge, end-to-end AI-assisted architecture for detecting APTs – RANK. We propose advanced algorithms and solutions for four consecutive sub-problems: 1) alert templating and merging, 2) alert graph construction, 3) alert graph partitioning into incidents, and 4) incident scoring and prioritization. Additionally, we discuss the necessary optimizations and techniques enabling the system to operate in a real-time fashion. We evaluate our architecture against the 2000 DARPA, Mordor, as well as a large number of real-world datasets from enterprise networks. Extensive results are provided showing four orders-of-magnitude reduction in the amount of data to be reviewed, innovative extraction and security-aware scoring of incidents. The extracted incidents can be further used for downstream tasks. In our experiments where we have access to a portion of alert labels, we are able achieve 87% balanced accuracy.