Self-Residual Deep Reinforcement Learning for Volt-Var Control in Active Distribution Networks

For inverter-based volt-var control in active distribution networks, we generally set the action space of DRL as the capabilities of those controllable devices. Deep reinforcement learning can acquire a near-optimal policy, while there is still a gap between optimal solutions. One of the reasons is that DRL cannot search optimal actions in the action space, and the large action space further worsen the problem. To alleviate the problem, we propose a self-residual DRL method to reduce the action space significantly in the second learning process. The first learning process is the same as the conventional actor-critic approach. The second self-residual process sets a smaller residual action space and learns the residual policy on the base policy learned in the first process. To avoid the final actions out of the action space, we design a dynamic method for the residual action space according to the action of the base policy. Self-residual DRL is compatible with any actor-critic approach, and we propose a self-residual soft actor-critic approach. Simulations are conducted on the 33 and 69-bus distribution networks that show the self-residual DRL approach can acquire a better VVC performance.