Reinforcement Learning Enhanced LQR and Control Lyapunov Functions for Spacecraft Proximity Operations

Spacecraft autonomy is a major barrier to increasing the scope, ambition, and affordability of both Earth-based and deep-space missions. Reinforcement learning (RL) offers huge potential in solving this problem, however, their adoption is hampered by the lack of stability guarantees, search space size and the complexity of spacecraft optimal control problems. Control techniques, such as control Lyapunov functions and linear quadratic regulators can help the RL frameworks find the optimal solution. The combination of these controllers with RL is investigated in Clohessy-Wiltshire–Hill dynamics. Several different greedy control approaches, as well as a novel nongreedy formulation, are considered for time-optimal and fuel-optimal transfers. Comparisons with optimal control theory, particle swarm optimisation and RL-only simulations are presented, demonstrating the effectiveness of RL-enhanced control approaches.