ASP-DRL: A Novel Framework for Unifying IoT Energy Usage Flexibilities Characterized by Neural Networks and Optimization Models

Managing energy usage flexibility has been identified as an effective way to coordinate Internet of Things (IoT) technologies and transition to a low-carbon future. However, the various models of these flexibilities may not be fully understood, and some may even be characterized by closed-box neural networks, such as those used for electric vehicles (EVs) charging. In this article, we propose a novel framework called augmented shadow-price deep reinforcement learning (ASP-DRL) for the online, distributed management of IoT under multiple sources of uncertainties, including renewable energy sources (RESs), wholesale electricity prices, and EV behavior patterns. To address these challenges, the proposed framework combines scheduling mechanisms for neural networks and optimization models to maximize total social welfare. Within the ASP-DRL framework, the policy network adaptively learns about system uncertainties and delivers actions to different distributed entities, either to form augmented objective functions for optimization models or to guide neural networks. We also present a distribution correction algorithm that enhances the vanilla soft Actor-Critic method with attention-based maximal corrective feedback, resulting in faster convergence and better performance. Our numerical studies demonstrate the superiority of the proposed ASP-DRL framework compared to conventional deep reinforcement learning (DRL) and optimization-based approaches.