Residual Deep Reinforcement Learning with Model-based Optimization for Inverter-based Volt-Var Control

A residual deep reinforcement learning (RDRL) based on an approximate-model-driven optimization approach is proposed for inverter-based volt-var control (IB-VVC) in active distribution networks. A modified Markov decision process is introduced to formulate the model-based and RDRL-based IB-VVC simultaneously, and then RDRL learns a residual action based on the action of the model-based approach with an approximate model. It inherits the control capability of the approximate-model-based optimization and enhances the policy optimization capability by residual policy learning. Since the approximate model acquired by operators is generally relatively reliable, the action solved by model-based optimization approaches is not far away from the optimal one. This allows RDRL to search for the residual action in a smaller residual action space, which further improves the approximation accuracy of the critic and reduces the search difficulties of the actor. Simulations demonstrate that RDRL improves the optimization performance considerably throughout the learning stage and verifies their three rationales for superior performance point-by-point on 69 and 141 bus balanced distribution networks.