Electric Vehicles Management in Distribution Network: A Data-Efficient Bi-Level Safe Deep Reinforcement Learning Method

In this paper, a deep reinforcement learning (DRL)-based electric vehicles (EVs) management strategy is proposed to achieve peak shaving and regulate the voltage violations in distribution networks. We present a new approach for modeling the EV user willingness considering the effect of multi-attribute attitudes and price incentives on user’ behaviors. The real-time optimal regulation problem is modeled as a bi-level optimization and then formulated as a constrained Markov decision process. To solve the problem, we first define an EV energy availability criterion for K-means clustering method to assess the EV scheduling potential for hierarchical dispatching, which facilitates meeting the user charging demand constraint. Then we present a bi-level safe soft actor critic algorithm with optimal convergence guarantee to solve the optimal sequential gaming problem between distribution system operator and charging stations aggregator. For each constraint, we employ individual neural networks to compute the cost Q-function and update independent Lagrange multiplier, which contributes to the setting of expected cost thresholds for practical application. In addition, we develop a data-enhanced method using Gaussian mixture model with a two-stage training process to improve the data efficiency and model robustness, as well as avoiding the constraint violations in the start-up process. Test results on the modified IEEE 33-bus and 118-bus systems with the real-world data are presented to validate the effectiveness of the proposed approach.