Coordinated Reactive Power Optimization for Transmission and Distribution System With Gross Prediction Errors: A Modified Belief Markov Decision Process-Based Reinforcement Learning Methodology

Given the significant uncertainty of distributed generations (DGs) in the transmission and distribution (T&D) system, we propose a novel Modified Belief Markov Decision Process-based (MBMDP-based) Reinforcement Learning scheme (namely, MBMRL) for Day-ahead Coordinated Reactive Power Optimization Problem (DCRPOP) with gross prediction errors. Firstly, we characterize DCRPOP as a Partially Observable Markov Decision Process (POMDP) model embedded with a belief state, which utilizes the probability distribution of the observed state with errors to portray the precise state. Secondly, the POMDP model is transformed into the MBMDP model by introducing the misestimated belief state probability vector. A misestimated belief state probability vector is incorporated into the belief state update process to enhance the confidence level in circumstances of a significant data discrepancy. Then, the MBMDP block with a high confidence level for the precise state is inputted into the underlying network architecture of the multi-agent actor-attention-critic algorithm, assisting agents in independently capturing features and outputting optimal decision-making actions even with significant data errors. Case studies are conducted in two T&D systems with different scales. The training dataset is constructed based on a real historical database from Suzhou, China. Simulation results validate the superior performance and scalability of the proposed methodology.