Using reinforcement learning for optimizing heat pump control in a building model in Modelica

With the increasing share of renewable energy sources in the electricity grid comes the need to exploit the available flexibility at the demand side. Demand response programs seek to exploit the flexibility of consumers by motivating endusers to shift demand based on grid signals. An important source of flexibility available at residential consumers are Thermostatically Controlled Loads (TCLs). Additionally, recent advances within the reinforcement learning area have made it possible to apply this technique to a large range of problems. Driven by these promising examples, a Batch Reinforcement Learning (BRL) algorithm is applied to a TCL. An important property of the complex optimization problem studied here, is its partial observability. The main contribution of this paper is the application of BRL to a detailed building and heating system model, implemented in Modelica. A detailed TCL model allows to perform an in-depth analysis of the effects of partial observability on the performance of the chosen control strategy. Ultimately, this paper illustrates that Modelica can be used to provide a detailed environment for a BRL algorithm. At the end, the learned control policy has been compared with two other control policies. The obtained policy outperforms both and is economically feasible after a limited amount of training days.