Prosumer Cost Efficiency and Ensuring Grid Stability Through a Hierarchical PPO Framework in Decentralized Community Energy Management

This research introduces a methodology based on reinforcement learning to address the cost reduction challenges faced by prosumers in a smart home-based community. The proposed method tackles the unpredictability of distributed energy resources (DERs) and the heterogeneity of prosumers by introducing Hierarchical Home Energy Management with a Noise-Adaptive Proximal Policy Optimization (H2EN-PPO) framework. This multi-agent problem is addressed using deep reinforcement learning using decentralized critic and decentralized value neural networks. The highest-tier node focuses on resolving the internal energy pricing predicament, while the lower nodes manage the scheduling of household appliances. To emulate human behavior accurately, distributed actual power transition data obtained from various domestic devices is incorporated, accounting for aspects such as environmental circumstances and and dynamic pricing in real-time. Real-world datasets are employed in simulations to evaluate the proposed approach. To maintain grid stability, a suitable reward function is implemented to prevent overloading of the transformer connected to a particular community. This reward function is also designed to address the variation in human behavior when it comes to electric vehicle (EV) charging, taking into account factors like range anxiety and time anxiety. Additionally, this function is capable of optimizing the utilization of the photovoltaic (PV) generation. The findings indicate that the proposed approach substantially minimizes prosumers’ daily costs, outperforming existing methods. The H2EN-PPO framework offers advantages in terms of cost reduction and efficient resource utilization in community energy trading scenarios. The contributions of this research lie in addressing the challenges of DER uncertainty and prosumer heterogeneity, as well as incorporating realistic human behavior models into the optimization framework.