A Comparison of the State-of-the-Art Reinforcement Learning Algorithms for Health-Aware Energy and Emissions Management in Zero-Emission Ships

Zero-emission ships (ZESs) have gained interest to comply with the stringent regulations of international maritime organization. One way to build ZES is the hybridization of fuel cells with batteries. Traditionally, for a newly built ship, the energy and emissions management system (EEMS) is designed based on the initial condition of the fuel cells and batteries and used with fixed parameters in future execution. However, for a fuel cell and battery ZES, the EEMS gradually becomes suboptimal since the characteristics of fuel cells and batteries are continuously changing due to aging and degradation. In this article, a reinforcement learning (RL)-based EEMS is developed such that it can learn and adapt continuously to changes in the fuel cell/battery characteristics. Within RL, different types of algorithms, such as double deep $Q$-learning (DDQL), soft actor–critic (SAC), and proximal policy optimization (PPO) are implemented. The results are benchmarked against those of a typical rule-based EEMS. Each RL algorithm is trained with four reward function formulations; negative cost ($r_{1}$), negative quadratic cost ($r_{2}$), inverse cost ($r_{3}$), and inverse quadratic cost ($r_{4}$). The results demonstrate that health-aware EEMS can minimize fuel consumption and component degradation costs. $r_{1}$ has led to the lowest operational expenses (OPEX) followed by $r_{2}$, while $r_{3}$ and $r_{4}$ have high OPEX. Among the three algorithms, the DDQL led to the lowest reward followed by the SAC and then the PPO, when trained with $r_{1}$ and $r_{2}$.