Offline Deep Reinforcement Learning-Based Home Energy Management Systems With Heterogeneous EV Charging Load Models

With increasing penetration of Electric Vehicles (EVs) into the transportation system and smart electricity grid, there is a growing need for integrating them into Home Energy Management Systems (HEMS). This integration within HEMS introduces dynamic user behaviors and time-varying charging demand, thus posing challenges for the HEMS. To mitigate these challenges, this paper proposes a charging model for heterogeneous EVs that covers the range of Plug-in Hybrid EVs (PHEVs), Range-Extender EVs (REEVs) and Battery EVs (BEVs) with/without heat pumps. The proposed heterogeneous EV charging model considers weather conditions, estimated mileage and driver’s experience to describe the dynamic charging demand and the anxiety level influencing their behavior. To optimize the HEMS operation, minimizing the energy cost and ensuring comfort, this paper introduces an offline Deep Reinforcement Learning (DRL) algorithm which learns directly from pre-collected datasets, avoiding the cost and safety issues associated with continuous real-world interactions. The algorithm incorporates the Huber loss and a Q-quantile estimator to mitigate performance degradation from dataset anomalies such as data noise, sensor failure and human error, resulting in more robust HEMS optimization strategies. Experimental results demonstrate the method’s effectiveness in reducing total costs and analyze the performance of household devices with two different electricity rates.