Adaptive Control of Practical Heat Pump Systems for Power System Flexibility Based on Reinforcement Learning*

Modern power systems are under flexibility shortage because of high renewable penetration. As heating systems are increasingly integrated with electric power systems, heat pumps have become a valuable source of power system flexibility. However, utilizing the flexibility of heat pumps necessitates additional regulation system on the heat pump, which complicates their design. Many commercially available heat pump systems modulate through a relatively slow ramping process and suffer from significant input transport delays. Due to complex dynamical process in heat pumps, a traditional model-free closed-loop power controller, such as the proportional-integral-derivative type, may result in poor transient performance. In contrast, an open-loop control may provide faster transient response at the expense of significant steady-state error. The steady state error is especially problematic due to high non-linearity of heat pump power consumption with respect to working condition variables, such as source and sink media temperatures and mass flow levels. This paper proposes an reinforcement learning based open-loop control system that provides fast transient response but is adaptive in nature to compensate for the non-linearities arisen from changing working conditions. The impact of working condition changes is captured with the trained deep neural network that modifies the modulation input to minimize potential steady-state power tracking error.