Anti-Jamming Attack Mixed Strategy for Formation Tracking Control via Game-Theoretical Reinforcement Learning

Communication, the basis for the unmanned aerial vehicle (UAV) to exchange information (e.g., displacement, velocity, or direction), plays a role in multi-UAV to perform formation tracking missions. In complex environments, UAV communication is often subject to jamming attacks, affecting the formation process. Therefore, studying the formation tracking control problem in jamming attacks is of great significance. Typically, the actions of the UAV consist of two fundamental modules: mobility strategy and communication strategy. In this paper, we design an anti-jamming attack mixed strategy for formation tracking control of the multi-UAV system. In practical scenarios, multi-UAV systems not only require the accomplishment of formation maneuvers but also necessitate effective mitigation of jamming attacks caused by other UAVs. Therefore, we suppose there are three types of UAVs: leaders, followers, and jammers. To illustrate the interactions between multiple UAVs of the formation tracking process under jamming attack, a three-layer Stackelberg game is constructed. Leaders and followers need to resist interference from jammers during the formation tracking process with a leader-follower structure. Leaders and followers achieve formation tracking through cooperation. The interactions between jammers and the other UAVs form a non-cooperative game. Moreover, the utility functions of a mixed strategy, which contain mobility and communication strategies, are designed for the three-layer Stackelberg game. The Stackelberg-Nash equilibrium of the designed game model is proven to exist. For seeking the Stackelberg-Nash equilibrium, a tri-level actorcritic (Tri-AC) reinforcement learning algorithm is designed. The convergence of the designed algorithm is also proved theoretically. Finally, the effectiveness of the designed method is verified by various simulation experiments.