Fast Real-Time Reinforcement Learning for Partially-Observable Large-Scale Systems

We propose a fast real-time reinforcement learning (RL) control algorithm for large-scale partially-observable linear dynamic systems. We first develop a one-shot RL method for designing model-free optimal controllers based on a finite-time history of the inputs and the outputs. However, when the system dimension is large, this method may suffer from a long learning time. To overcome this problem, in the second half of the paper we introduce a new notion of approximation to the design, where the original set of input-output history is replaced by a much shorter set. We show that this approximation can lead to a nearly optimal controller that is based on a lower-dimensional approximant of the original system in terms of reachability and observability. We provide a guideline for determining an appropriate length of the input-output history to reduce the suboptimality gap. The dimension of the resulting suboptimal controller is far less than that of the optimal controller, thereby speeding up learning time. The learned controller, however may cause instability when implemented in the original high-dimensional system by adversely exciting the approximation error. We theoretically establish the conditions for closed-loop stability using robust control theory, followed by numerical investigations of the trade-offs between learning time, length of input/output history, and closed-loop performance. The effectiveness of the method is illustrated using examples from electric power systems, modeled by partially-observable nonlinear differential-algebraic equations.