Stochastic Path Planning in Partially Observable Environments via Temporal-Difference Learning

In this study, we address the critical challenge of stochastic path planning in partially observable environments with inherent uncertainties, a problem central to ensuring the safety and effectiveness of autonomous systems in high-stakes scenarios such as planetary exploration. Temporal difference learning (TD) methods have demonstrated considerable effectiveness for planning in environments with Markov Decision Processes (MDPs) as the underlying model. To extend these capabilities to partially observable settings, we build upon an existing Monte Carlo tree search method, POMCP, and introduce our novel algorithm, TDPOMCP. This algorithm adopts TD methods to effectively tackle partially observable MDPs (POMDPs). Specifically, TDPOMCP integrates expected state-action-reward-state-action (ESARSA) and Q-learning methods to calculate both risk-averse and safe optimal paths. To validate the efficacy of our proposed approach, we conduct numerical simulations using a Mars rover traversing a two-dimensional map populated with predetermined obstacles.