A Probabilistic Online Policy Estimator for Autonomous Systems Planning and Decision Making

Partially Observable Markov Decision Process (POMDP) models are a popular probabilistic modeling method for continuous planning in systems that operate in partially observable, uncertain environments. This paper presents an online algorithm, N-Step Look-Ahead, for solving POMDP models with specific characteristics: flexible state-space, dynamic model parameters (e.g. probability distributions), and real-time constraints (e.g. response needed in fractions of seconds). This algorithm computes the best executable policy by creating a belief-tree starting from the current belief state and employing a look-ahead search over a finite time horizon. To address time-accuracy trade-offs, various heuristics in combination with a distance-based clustering technique is employed to expand and explore only high value belief nodes. To evaluate the accuracy of our algorithm, we compare the online policies computed from N-Step Look-Ahead with offline policies calculated using a customized Q-learning algorithm. We show that the online algorithm can compute optimal policies for beliefs, where belief probabilities are not normally distributed, by looking only a few steps ahead in the belief tree. We discuss computing online policies for normally distributed belief states using our algorithm and explain why they can be different from that obtained from the offline algorithm. Finally, we show how useful heuristics can be developed from Q-learning results to improve the N-Sep Look-Ahead in terms of computation time and accuracy, especially for large POMDP models.