Scalable Autonomous Separation Assurance With Heterogeneous Multi-Agent Reinforcement Learning

In this article, a scalable autonomous separation assurance framework is proposed for high-density en route airspace sectors with heterogeneous aircraft objectives. To handle the complex dynamic decision making under uncertainty, multi-agent reinforcement learning is used in a decentralized approach with each aircraft being represented as an agent. Based on this, each agent locally solves the separation assurance problem, allowing the framework to scale to a large number of aircraft. In addition, each agent has the ability to learn the intention of the intruder aircraft, which is essential in environments with heterogeneous agents. Numerical experiments are performed in a real-time air traffic simulator. The results demonstrate that the proposed framework is able to effectively ensure the safe separation of heterogeneous agents, while also optimizing the intrinsic agent objectives in high-density en route airspace sectors. In addition, the efficiency of the proposed framework is demonstrated and shown to provide real-time decision making for separation assurance. Note to Practitioners—In commercial aviation, the workload of human air traffic controllers increases with the growth of air traffic density. Robust decision making systems to augment human air traffic controllers allows for increased air traffic without increased workload, resulting in a safer airspace environment. In addition, advanced air mobility (AAM) is concerned with low-altitude airspace operations with both human and autonomous pilots. In this environment, autonomous real-time separation assurance systems are required. Most traditional separation assurance approaches fail to handle stochastic and high-density environments, rendering them inapplicable to future high-density traditional airspace and the envisioned low-altitude AAM operations. Therefore, it is important to study decentralized approaches that can place the separation assurance problem as an intrinsic objective of each aircraft to ensure cooperation in high-density airspace. With this consideration, a scalable, decentralized autonomous separation assurance framework capable of handling heterogeneous agents is proposed in this article. This framework is able to perceive the current air traffic environment and select speed advisories to ensure safe separation requirements, while balancing intrinsic objectives such as minimizing delay. While one limitation of multi-agent reinforcement learning is the long training time, this article demonstrates how the framework also can leverage modern computing clusters to significantly reduce training time without sacrificing performance.