Multi-agent Deep Reinforcement Learning for Spectrum and Air Traffic Management in UAM with Resource Constraints

The quest to optimize frequency utilization in aviation dates back to the very beginning of commercial flight. Over time, spectrum management techniques and new radio technologies have been developed and implemented to assist in meeting the growing demand placed on the system by increased airspace operations. A new challenge to existing aviation spectrum capacity and utilization is emerging with the development of new Urban Air Mobility operations. These new entrants to the airspace will place additional demands to an existing scarce resource, and a new approach to the way frequency is used and managed is needed. This paper investigates the use of deep reinforcement learning techniques to optimize airspace operations efficiency (i.e., minimize the average mission completion time) and safety while considering constrained resources. Through jointly managing spectrum allocation, vehicle scheduling and flight speeds, a multi-agent system using Value Decomposition Network (VDN) collaborates to optimize channel utilization, flight time and departure wait times. To minimize mission completion time, a multi-stage Markov Decision Process (MDP) is formulated where states such as frequency channel availability, uplink signal to interference plus noise power ratio, aircraft location and flight status are jointly considered. A model-free MDP utilizing temporal difference learning and gradient descent is used to train and optimize weight parameters of a three layer deep neural network. Finally, using modeling and simulation the proposed VDN solution is evaluated against two established baseline solutions, namely, the heuristic greedy algorithm (HGA) and orthogonal multiple access (OMA).