Multi-Agent Deep Reinforcement Learning for Simulating Centralized Double-Sided Auction Electricity Market

Developing optimal bidding strategies for the market participants plays a crucial role in increasing the profit of the electricity market. For the complex double-sided auction market featured with partial observability, uncertainty and dynamic nature in the bidding transaction, current methods like Reinforcement Learning (RL) based on single agent framework are difficult to model the strategic bidding behavior of both supply side and demand side with elastic demand. For this purpose, this paper proposes a multi-agent system based on Multi-Agent Deep Deterministic Policy Gradient (MADDPG), combined with Prioritized Experience Replay (PER) mechanism, namely MADDPG-PER, to simulate the complex market environment. The multi-agent system based on MADDPG-PER is not only able to deal with the sparse reward gradient phenomenon for auction-based electricity market, but also can find superior bidding strategies for the market participants. The MADDPG-PER algorithm is evaluated on 9- bus and 30- bus congested network, where both supply and demand sides are modeled as RL agents. The results demonstrate that MADDPG-PER outperforms the state-of-the-art methods under different level of system uncertainty.