Defending Against APT Attacks in Cloud Computing Environments Using Grouped Multiagent Deep Reinforcement Learning

Advanced persistent threats (APTs) pose a significant challenge to cloud computing security in the evolving landscape of cyber threats. Traditional defense models rely heavily on the attacker’s historical attack information, which greatly limits the effectiveness of actually dealing with APT attacks. To address this issues, we investigate an attack-defense game model in clouding computing environments, where multiple attackers and multiple defenders are supposed to compete for resource allocation on the cloud servers. In order to develop more effective defense strategies, we formulate the optimization problem to maximize the average rewards of defenders under constraints of the maximum available resource and acceptable cost. To solve this, we propose to use the multiagent deep reinforcement learning (RL) method to cope with the high uncertainty and dynamics of attack behavior. Then it is proposed to divide all defenders into cooperative groups and allow defenders within each group can jointly optimize the defense strategy through sharing information and experience. On this basis, we propose a novel grouped multiagent deep RL defense (GMADRLD) algorithm, which can effectively mitigate the issue of state space explosion while achieving good defense effect. Simulation results not only demonstrate the effectiveness of the proposed GMADRLD algorithm in dealing with the attacker’s ever-changing strategies, but also show that it is able to strike a balance between defense performance and computational complexity.