Online Volt-Var Control Based on Reinforcement Learning in Active Distribution Systems Considering PV Inverters

Due to the wide deployment of photovoltaic (PV) power generation, voltage stability is facing a severe challenge in the active distribution networks (ADNs). Reinforcement Learning (RL) techniques have been widely adopted in the power system optimal operation and quick control with high-performance achievements. Existing RL algorithms can handle only homogenous control action spaces, i.e., either continuous or discrete actions. However, the voltage control problem has hybrid action spaces. This paper proposes a hybrid soft actor-critic (H-SAC) algorithm for online quick-optimal voltage control. First, the problem is formulated to consider all types of control actions, including the setpoints of PV inverters. Second, it is standardized to be compatible with RL techniques. Third, the fundamental sample-efficient entropy-based maximization SAC algorithm is enhanced to train a policy with hybrid action spaces, where an action selection method is employed to map the continuous output of the actor-network into the integer action spaces. The proposed method is applied to the modified IEEE 33-bus system, where the voltage has been controlled with a very low computational burden.