A DRL-Based Hierarchical Game for Physical Layer Security with Dynamic Trilateral Coalitions

In this paper, a novel hierarchical game framework for physical layer security (PLS) with dynamic trilateral coalitions is studied. In the considered system, legitimate users (LUs) aim to transmit secret data to associated base stations (BSs) via uplink communications under the threat of eavesdroppers (EVs), while there also exists jammers (JAs) which may choose to form coalitions with either LUs for increasing their secrecy transmission rates or EVs for increasing their eavesdropping rates in exchange for potential rewards. Different from the existing work, we explore such complicated while dynamic coalition relationships under the uncertainties of wireless systems (e.g., time-varying channel conditions), and formulate a hierarchical game integrated with a dynamic trilateral coalition formation game to model the strategic interactions among all three parties, i.e., LUs, JAs and EVs, in PLS. Particularly, we first analyze stability conditions of the trilateral coalitions. On top of this, we further propose a deep reinforcement learning (DRL) based approach for reaching the equilibrium with long-term performance guarantees for the hierarchical game. Simulations evaluate the proposed solution and show its superiority over counterparts.