NestedGNN: Detecting Malicious Network Activity with Nested Graph Neural Networks

Network attacks are dramatically increasing over the years. A graph can accurately model the network activities. Therefore, graph-based techniques are frequently used to detect network threats. Motivated by the strong representation of graph neural networks (GNNs), many GNN-based techniques have been proposed for various security problems, such as network threat detection, malware detection, insider threat detection, and fraud detection. Most GNNs work on the classical attributed graph structure, while we observe that a nested graph structure is a more accurate representation for modelling enterprise network, where the communications between hosts form a graph, while the local activities of each host, e.g., local event graph, form an inner graph. Observing no existing GNNs can directly learn on such a nested graph, in this paper, we designed NestedGNN, the first graph neural network for nested graphs. NestedGNN consists of three layers, i.e., inner GNN layers, nested graph layers, and outer GNN layers. We successfully applied it to compromised host detection. NestedGNN can significantly improve the performance over traditional methods on a publicly available cybersecurity dataset.