Convex synthesis of randomized policies for controlled Markov chains with density safety upper bound constraints

The main objective of this paper is to synthesize optimal decision-making policies for a finite-horizon Markov Decision Process (MDP) while satisfying a safety constraint that imposes an upper bound on the state probability density function (pdf) of the underlying Markov Chain (MC) for all time steps. The classical approach based on state-action frequencies for constrained MDPs yields decision policies that provide safety constraint satisfaction for stationary distributions (i.e., asymptotically), but not necessarily providing safety during the transient regime. This paper introduces a new synthesis method for randomized Markovian policies for finite-horizon MDPs, where the safety constraint satisfaction is guaranteed for both the transient and the stationary distributions independent from initial state (i.e., providing safe policies for the worst-case analysis). An efficient Linear Programming (LP) based synthesis algorithm is proposed, which produces a convex set of feasible policies and ensures that the expected total reward is above a computable lower-bound. A simulation example of a swarm of autonomous agents is also presented to demonstrate the practical importance of having safe policies.