G-Learning: Equivariant Indirect Optimal Control with Generating Function

This paper presents a new formulation of data-driven, learning-based optimal control with the Hamilton-Jacobi theory. In contrast to the common practice of rein-forcement learning for dynamical systems, where the control policy is parameterized by deep neural network and the control parameters are optimized directly, we propose to adopt the indirect optimal control where the necessary conditions for optimality are first constructed by Pontryagin's minimum prin-ciple. Then, the resulting two-point boundary value problem is solved by learning the generating function associated with the optimality conditions that are considered as a Hamiltonian system. Further, it is shown that the sampling efficiency can be improved when there exists an invariance in the dynamics. The foremost benefit is that this provides a set of optimal controls for varying boundary conditions, which cannot be systematically addressed in the policy gradient.