Using Cosimla Within Policy Iteration for Mdps with Large State Spaces

Classical policy iteration methods to solve Markov Decision Processes (MDPs) incur a computational complexity that critically depends on the size of state space. Such computational complexity can be prohibitive when the size of state space is enormous or even countably infinite. To improve the computational effectiveness, we develop a computational method that strategically integrates policy iteration and a recently developed approach called COSIMLA (Combining Numerical Linear Algebra and Simulation). We provide analysis for the proposed computational method and demonstrate its comparative advantages through numerical experiments.