Reinforcement Learning Heuristics for Aerospace Control Systems

Reinforcement learning (RL) is a form of machine learning (ML) where an agent is directed to a goal by learning from interactions within an environment. RL has been used to solve previously intractable problems such as Backgammon, Go, and Starcraft and offers a promising class of solutions to complex problems in various domains, including ones involving air and space. One of the major hindrances of using RL, more specifically deep RL (DRL), as a method is the variability with which an agent trains, which results from algorithm hyperparameters (e.g., batch size, training epochs, learning rate, etc.), deep neural network architectures (e.g., number of inputs, outputs, hidden layers and neurons in each layer, activation functions, connectivity, etc.), goal conditions, and reward functions. A slight modification to any of these factors can have a major impact on the training of the agent and the effectiveness of the learned policy. As such, hyperparameter and architecture searches are common practice in other domains involving neural networks. This work focuses on a systematic approach to DRL hyperparameter and architecture optimization for aerospace control systems using direct search, zeroth-order optimization, Bayesian optimization, and asynchronous hyperband-based training methods that are demonstrated in two different safety-critical aerospace environments and tasks, shedding light on general rules of thumb for RL applied to aerospace control systems.