Distributed Deep Deterministic Policy Gradient Agents for Real-Time Energy Management of DC Microgrid

This paper presents a real-world military use of a microgrid with Vehicle-to-Grid (V2G) and Vehicle-to-Vehicle (V2V). This plug-and-play system delivers efficient, fast-deploying power for a contingency base within 20 minutes, generating up to 500 kW of three-phase power. It utilizes vehicle-based Internal Combustion (IC) engine generators and an Energy Storage System (ESS) to power vehicle equipment and off-board loads like shelters and communication centers. However, fluctuating off-board loads create operational challenges for this V2G-V2V microgrid. This work proposes a real-time implementation of energy management based on distributed Deep Deterministic Policy Gradient (DDPG), a deep reinforcement learning, to address this problem for continuous state and continuous action spaces. This work formulates the energy management problem as a Markov decision process, considering random load demands and fuel costs. This paper aims to minimize the microgrid's operating costs by optimizing the dispatch of onboard vehicle power generators and storage units. With simulation experiments, this work demonstrates the effectiveness of the proposed distributed deep reinforcement learning approach. The results show the distributed DDPG agents can utilize the state of the ESS and significantly decrease the microgrid's overall operation costs. This validates the practical value of distributed DDPG for economic operations in military microgrids.