Reinforcement Learning for the Agile Earth-Observing Satellite Scheduling Problem

This work explores reinforcement learning (RL) for on-board planning and scheduling of an agile Earth-observing satellite (AEOS). In this formulation of the AEOS scheduling problem, a spacecraft in low Earth orbit attempts to maximize the weighted sum of targets collected and downlinked. The AEOS scheduling problem is formulated as a Markov decision process (MDP) where the number of upcoming imaging targets included in the action space is an adjustable parameter to account for clusters of imaging targets with varying priorities. Monte Carlo tree search (MCTS) and supervised learning are used to train a set of agents with varying numbers of targets in the action space. Two backup strategies are explored for MCTS—an incremental averaging operator and a maximization operator. For all backup operators, performance asymptotically increases as the number of targets in the action space approaches the maximum number of available targets. A benchmark is computed with MCTS to determine an upper bound on performance. Furthermore, MCTS is compared with solutions generated by a genetic algorithm. MCTS demonstrates a 2%–5% increase in average reward at 10%–20% of the single-core wall clock time of the genetic algorithm. A search of various neural network hyperparameters is presented, and the trained neural networks are shown to approximate the MCTS policy with three orders of magnitude less execution time. Finally, the trained agents and the genetic algorithm are deployed on varying target densities for comparison purposes and to demonstrate robustness to mission profiles outside of the training distribution.