On Exact Embedding Framework for Optimal Control of Markov Decision Processes

This article deals with the embedding framework of Markov decision processes (MDPs) with discrete state and action space to find optimal actions. The optimal control problem of MDPs can be efficiently tackled by restructuring the same into an equivalent linearly-solvable Markov decision processes (LMDPs) through the method called embedding. However, state costs under the embedding may not exactly match the original costs and even assume unrealistic values. In this work, we derive a constructive sufficient condition to devise an exact embedding solution rendering the embedded state cost to match the original system. Furthermore, since, in this case, the embedding implies a transition from the discrete to continuous action space, the correlation between the obtained continuous action and an equivalent desired discrete action is investigated using a maximum a posteriori probability-based method. Finally, some examples, including mammalian cell-cycle network, are presented to demonstrate the effectiveness of the proposed method.