Multi-Period and Multi-Spatial Equilibrium Analysis in Imperfect Electricity Markets: A Novel Multi-Agent Deep Reinforcement Learning Approach

Previously works on analysing imperfect electricity markets have employed conventional game-theoretic approaches. However, such approaches necessitate that each strategic market player has full knowledge of the operating parameters and the strategies of its rivals as well as the computational algorithm of the market clearing process. This unrealistic assumption, along with the modeling and computational complexities, renders such approaches less applicable for conducting practical multi-period and multi-spatial equilibrium analysis. This paper proposes a novel multi-agent deep reinforcement learning (MA-DRL) based methodology, combining multi-agent intelligence, the deep policy gradient (DPG) method, and an innovative long short term memory (LSTM) based representation network for optimizing the offering strategies of multiple self-interested generation companies (GENCOs) as well as exploring the market outcome stemming from their interactions. The proposed approach is tailored to align with the nature of the examined problem by posing it, for the first time, in multi-dimensional continuous state and action spaces, enabling GENCOs to receive accurate feedback regarding the impact of their offering strategies on the market clearing outcome, and devise more profitable bidding decisions by exploiting the entire action domain, and thereby facilitates more accurate equilibrium analysis. The proposed LSTM-based representation network extracts discriminative features which further improves the learning performance and thus promises more profitable offerings strategies for each GENCO. Case studies demonstrate that the proposed method i) achieves a significantly higher profit than state-of-the-art RL methods for a single GENCO's optimal offering strategy problem and ii) outperforms the state-of-the-art equilibrium programming models in efficiently identifying an imperfect market equilibrium with/without network congestion. Quantitative economic analysis is carried out on the obtained equilibrium.