3-D Autonomous Entry Trajectory Planning via Hybrid Action Reinforcement Learning

In this article, by employing a modular strategy and hybrid action reinforcement learning (RL), an intelligent 3-D entry trajectory planning approach is proposed to achieve online autonomous trajectory generation for any initial location and target within a large reachable area. To reduce training difficulty and enhance task adaptability, the modular design approach decomposes the entry trajectory problem into the bank and angle of attack (AOA) modules. The environment, with complete 3-D state space and entry dynamics, is established to train both modules, in which additional performance requirements are incorporated in the reward design, such as suppressing long-period oscillations and reducing reversal numbers. In the bank module, discrete-continuous hybrid action space is adopted to formulate the bank angle decision problem, which determines the sign and amplitude of the bank angle simultaneously, effectively avoiding the tradeoff between RL exploration difficulties and trajectory accuracy. Specifically, dueling hybrid twin delayed deep deterministic policy gradient algorithm is proposed to address the hybrid action problem and hence to train the 3-D bank module. Subsequently, the AOA module is trained to adjust the AOA profile to meet flight capability requirements. The simulations highlight the ability of our proposed algorithm to achieve autonomous robust trajectory planning in expansive reachable areas.