Reinforcement Learning for Linear Optimal Control Using a Modified Newton-Kleinman Method

In this paper, a modified policy iteration (MPI) online learning scheme is proposed for the infinite-time horizon control problem of a linear time-invariant system. Policies for continuous-time, continuous-space systems, especially when utilizing approximations, often lead to high-dimensional algebraic Riccati equations. Traditional direct solution methods, relying on eigenvector calculations, become impractical when dealing with high-order models. We develop an iterative approach specifically tailored to leverage the distinctive features of such high-dimensional problems. For the iterative algorithm, the convergence to the analytical optimal control solutions is validated. Furthermore, reinforcement learning techniques are integrated with MPI, to enhance the adaptability of the algorithm in practice, providing further optimization in solving these complex problems.