Deep-Shield: Multiphase Mitigation of APT via Hierarchical Deep Reinforcement Learning

With rapid development of industrial internet, industrial cyber-physical systems (ICPSs) have been widely deployed to perform and supervise industrial applications. However, ICPSs still face significant cybersecurity challenges. Traditional defense mechanisms are mostly static and passive, which may fail to provide real-time protection. To solve the aforementioned problem, moving target defense (MTD) technique has been proposed as a proactive solution. However, due to the increasing sophistication and persistence of cyberattacks, it is difficult for single-phase MTD approaches to provide effective defense by only mitigating individual phases of the attack process. Therefore, we present Deep-Shield, a novel multiphase MTD approach based on hierarchical deep reinforcement learning (HDRL) to improve the defense performance of single-phase MTD approaches when facing advanced persistent threat (APT) in ICPSs. We consider three representative MTD countermeasures, i.e., IP address shuffling, software diversity, and components redundancy, which are able to mitigate attacks at different phases of cyber kill chain and protect assets contained in critical infrastructures. First, we formulate the dynamic implementation of multiphase MTD countermeasures as a semi-Markov decision process. Second, we detect current attack patterns by a neural network called PerNet, which is derived from Anomaly Transformer. Then, we design a HDRL-based multiphase MTD algorithm for defense decision-making. Finally, through extensive experiments on a platform of software defined networks, we show that our proposed approach can achieve better defense performance compared with state-of-the-art solutions when dealing with APT.