Deep Differentiable Reinforcement Learning for Generic Energy Storage Day-Ahead Bidding with Transformer Neural Networks

The increasing prevalence of renewable energies has led to greater volatility in electricity prices, posing a greater challenge for Energy Storage (ES) to arbitrage effectively in the fluctuating spot markets. To better cope with price uncertainties using large amounts of hidden experiences from global electricity markets, we propose an approach called Transformer-DDRL to enhance the performance of Day-Ahead (DA) ES bidding. Initially, based on the Transformer network, we develop an expressive deep neural network that is capable of encoding large amounts of bidding experiences and generating 24-hour DA bidding curves. Subsequently, we train the proposed Transformer network based on Deep Differentiable Reinforcement Learning (DDRL). Using DDRL, we directly optimize the network parameters of the Transformer with respect to the bidding objective to achieve effective learning. Finally, we train the proposed bidder on a global price dataset and achieve state-of-the-art returns in multiple markets, demonstrating its promise as a versatile market bidding model.