Safe Q-Learning Approaches for Human-in-Loop Reinforcement Learning

In this work, we formulate a safe human-in-loop reinforcement problem by allowing human experts to specify safe actions during the training. We propose two algorithms, safe Q-learning and partially safe Q-learning, which use constrained action spaces during training to find a safe optimal policy. In the case of partially safe Q-learning, the concept of safety ratio is introduced to provide the agent an ability to explore while safety is guaranteed. The proposed algorithms are corroborated by performing simulation studies for four different environments of varying complexity. It is shown that a partially safe Q-learning approach outperforms safe Q-learning and Q-learning on various tasks.