Automatic Generation Control Based on Brain-Inspired Continuous Return Distribution Algorithm

The increased penetration of renewable energy has triggered significant random disturbances that affect the performance of automatic generation control, posing a challenge to the stability of novel power systems. Reinforcement learning based on Markov stochastic processes has advantages in obtaining stochastic optimal solutions and has gradually been explored and applied by scholars to automatic generation control. Among these, the algorithms based on the Q learning framework are dominant. However, Q learning and its derivative algorithms suffer from the problem of Q value overestimation due to the discrete action space, which will be continuously magnified in high-dimensional space, leading to suboptimal solutions of the algorithm. Meanwhile, the training instability caused by improper gradient boundary setting of the algorithm, resulting in slow convergence. Therefore, this paper proposes a brain-inspired continuous return value distribution algorithm to obtain the optimal coordination of automatic generation control in a multi-area power grid. The proposed algorithm models the continuous return value as a Gaussian distribution through brain-inspired distributed learning and uses the variance of the value function to solve the problem of the Q value being continuously amplified in high-dimensional space. At the same time, a variance-based critical gradient adjustment strategy is introduced to replace fixed boundary clipping to avoid instability caused by improper boundary setting of the algorithm. The effectiveness of the proposed algorithm is verified by simulating a two-area automatic generation control system and a four-area automatic generation control system based on the Central China Power Grid. Compared with other reinforcement learning algorithms, it has better control performance, smaller frequency deviation, and faster convergence speed. Note to Practitioners—This paper proposes a novel model-free reinforcement learning algorithm for automatic generation control in novel power systems. The proposed algorithm overcomes the insufficient control accuracy and instability of existing reinforcement learning algorithms in novel power systems. After pre-training, it can be put into use in real time to maintain the frequency stability of power systems under strong random disturbances.