Chance-constrained formulation of MDPs under total reward criteria: an application to advertisement model

We consider a constrained Markov decision process (CMDP) with total reward criterion under random reward and cost parameters, and known transition probabilities. The CMDP problem is formulated as a joint chance-constrained Markov decision process (JCCMDP) which captures the situation where the decision maker is interested in the payoff function that is obtained with certain confidence and the random constraints are jointly satisfied with a given probability level. When the reward and cost vectors follow elliptically symmetric distributions and dependence among constraints is driven by a Gumbel-Hougaard copula, we show that the upper and lower bounds to the optimal value of the JCCMDP problem is given by the optimal values of two second-order cone programming problems. As an application, we consider a budget optimization of the advertising platforms and perform numerical experiments on randomly generated instances.