Optimization Strategy of EVA Participating in Peak Shaving Using Deep Reinforcement Learning

As an actual market entity, electric vehicle aggregators (EVA) play the role of providing ancillary services for the utility and optimizing the allocation of flexible resources to balance the economic interests of various market entities and inject stability into the grid. Instead of expanding the conventional model-driven solution, this paper proposes a data-driven deep reinforcement learning (DRL)-based optimization strategy for EVA. The proposed scheme is a two-layer energy management strategy, with the upper layer aiming to minimize peak-valley difference and maximize EVA revenue, and the lower layer aiming to minimize the cost of electric vehicle (EV) users. The energy trade between EVA and the utility and the charging/discharging behavior of EVs are stated as Markov decision. To obtain continuous power output and escape from training collapse, a proximal policy optimization (PPO) algorithm with a clipping mechanism is used to optimize the decision process. Moreover, multi-agent reinforcement learning is applied to the coordination and optimal scheduling of multiple electric vehicles. Compared with the deep Q-learning- and deep deterministic policy gradient-based investigations, the proposed method demonstrates superiority in stability and accuracy.