Large-scale Centralized Air Conditioner Scheduling with Ice Storage under Real-time Pricing by Curiosity-driven TD3 Algorithm

Nowadays, real-time price (RTP) is an effective approach to cope with the increasingly uncertain renewables, which can guide demand-side behavior to achieve peak shaving and valley filling. In modern cities, district cooling systems (DCSs) are considered ideal resources providing demand response, because they account for a large share of electricity and connect with multiple buildings with large thermal inertia. Especially, thermal energy storage in DCSs can be controlled strategically through storing/melting ice to maximize cost savings, following dynamic RTP. However, it is challenging to obtain an accurate prediction of future cooling demands and RTP, which further complicates the strategic real-time control. This study proposes a novel curiosity-driven reinforcement learning (RL) approach to realize real-time optimal control of thermal energy storage (i.e., ice storage) in DCSs. Firstly, the proposed approach introduces an intrinsic curiosity as compensation to conventional RL, which can facilitate RL explorations and improve training efficiency. Moreover, coupled with a long short-term memory network module, the proposed approach can further capture the future trends of the time-series data, e.g., demands and RTP. Finally, simulations based on a real-world DCS in Hengqin, China, demonstrate that the proposed algorithm achieves a 20.12% reduction in daily energy cost compared to operation without ice storage, and performs closely to the optimal benchmark.