Trajectory-Based Automata Learning for Offline Reinforcement Learning

Offline reinforcement learning (RL) learns a policy from a fixed batch of data. However, the overestimation of the values rooted in the out-of-distribution actions limits the applicability of offline RL. It results in methods constraining or regularizing the learned policy based on the dataset. Hence, offline RL algorithms try to tackle this issue by adding a secondary component, e.g., new hyperparameters or generative models. Our proposed method aims to tackle this problem by taking a new perspective using two approaches on the offline RL. We tend to use deterministic finite automoton (DFA) to learn Offline-DFA (offline deterministic finite automaton) or ARM-DFA (association rule mining DFA) from the trajectories in the dataset without implementing any secondary component or constraining or regularizing the learned policy. This means that the learned policy does not face out-of-distribution actions. We use the Offline-DFA (or ARM-DFA) to guide the policy learning process based on the dataset’s trajectories. We propose a novel method called automata learning for offline RL with q-learning (ALOQ), which implements this perspective into the offline RL. We show the convergence guarantee of our proposed method to an optimal policy. In practice, Offline-DFA (or ARM-DFA) is used to learn the ground truth task, i.e., encoded by a reward machine, which allows the agent to learn an optimal policy. We demonstrate the performance of our proposed method by comparing it to the existing offline RL methods, converging to an optimal policy faster than the existing offline RL methods.