On the Convergence of Policy Gradient for Designing a Linear Quadratic Regulator by Leveraging a Proxy System

Policy gradient algorithms have been shown to converge to the optimal controller in a linear quadratic regulator (LQR) design problem. Calculating policy gradients using the true system such as for a robot may, however, be costly. We consider a formulation in which an ample number of gradient calculations from a proxy or approximate system such as a simulator are mixed with gradient calculations from the true system at a few time steps. Under the simplifying assumption of exact gradient calculation, we show that policy gradient in this framework can still converge to a neighborhood of the desired optimal controller. In this sense, it can reduce both the burden and cost of experimenting only on the true system and potentially obtain a less conservative controller than a purely robust control approach that relies only on the proxy system for designing the controller.