Tractable Reinforcement Learning for Signal Temporal Logic Tasks With Counterfactual Experience Replay

We investigate the control synthesis problem for Markov decision processes (MDPs) with unknown transition probabilities under signal temporal logic (STL) specifications. Our primary objective is to learn a control policy that maximizes the probability of satisfying the STL task. However, existing approaches to STL control synthesis using reinforcement learning encounter a significant exploration challenge, particularly when expanding the state space to incorporate STL tasks. In this letter, we propose a novel reinforcement learning algorithm tailored for STL tasks, addressing the exploration difficulty by effectively leveraging counterfactual experiences to expedite the training process. Through experiments we show that these generated experiences enable us to fully employ the knowledge embedded within the task, resulting in a substantial reduction in the number of trial-and-error explorations required before achieving convergence.