Co-Optimization of Adaptive Cruise Control and Hybrid Electric Vehicle Energy Management with Clutch Engagement Decision Control via Reinforcement Learning

The co-optimization of adaptive cruise control (ACC) and energy management strategy (EMS) can improve the energy-saving capability of a hybrid electric vehicle (HEV). Co-optimization for a series-parallel HEV involves continuous control variables, such as the engine speed and torque, and discrete control variables, such as the clutch engagement decision. To solve the mixed-integer optimal control problem, we propose a hybrid-action reinforcement learning (HARL) algorithm called parameterized proximal policy optimization (PA-PPO) for series-parallel HEV co-optimization of ACC and EMS to minimize the clutch engagement/disengagement times, energy consumption, and guarantee the car-following performance. Firstly, the ACC model and the control-oriented HEV EMS model considering the clutch engagement decision control are constructed and combined to form the co-optimization model. Secondly, the PA-PPO algorithm is proposed and trained to solve the co-optimization problem. Compared to model predictive control (MPC), the proposed PA-PPO algorithm achieves similar fuel consumption while significantly reducing clutch engagement/disengagement times and jerk values. In addition, the computation time of PA-PPO is five-orders-of-magnitude smaller than MPC. Finally, the PA-PPO control performance is validated using a Simulink high-fidelity model (HFM).