Simulation-Based Sets of Similar-Performing Actions in Finite Markov Decision Process Models

Markov decision process (MDP) models have been used to evaluate the performance of policies in various domains, such as treatment planning in medical decision making. However, in practice, decision makers may prefer other strategies that are not statistically different from the actions in their initial policy of interest. To allow for decision makers' expertise and provide flexibility in implementing policies, this paper introduces a new framework for identifying sets of similar-performing actions in finite MDP models. These sets are obtained by combining a simulation-based dynamic programming algorithm for policy evaluation with a simulation-based statistical multiple comparisons procedure. The framework in this paper is applied in a medical decision-making setting to find sets of similar-performing antihypertensive treatment choices for a set of clinically representative patient profiles.