Learning in Constrained Markov Decision Processes

We consider reinforcement learning (RL) in Markov decision processes in which an agent repeatedly interacts with an environment that is modeled by a controlled Markov process. At each time step
$t$
, it earns a reward and also incurs a cost vector consisting of
$M$
costs. We design model-based RL algorithms that maximize the cumulative reward earned over a time horizon of
$T$
time steps while simultaneously ensuring that the average values of the
$M$
cost expenditures are bounded by agent-specified thresholds
$c^{\text{ub}}_{i},i=1,2,\ldots,M$
. The consideration of the cumulative cost expenditures departs from the existing literature, in that the agent now additionally needs to balance the cost expenses in an online manner while simultaneously performing the exploration–exploitation tradeoff that is typically encountered in RL tasks. This is challenging since the dual objectives of exploration and exploitation necessarily require the agent to expend resources. In order to measure the performance of an RL algorithm that satisfies the average cost constraints, we define an
$M+1$
dimensional regret vector that is composed of its reward regret, and
$M$
cost regrets. The reward regret measures the suboptimality in the cumulative reward while the
$i$
th component of the cost regret vector is the difference between its
$i$
th cumulative cost expense and the expected cost expenditures
$Tc^{\text{ub}}_{i}$
. We prove that the expected value of the regret vector is upper-bounded as
$\tilde{O}(T^{2\slash 3})$
, where
$T$
is the time horizon, and
$\tilde{O}(\cdot)$
hides factors that are logarithmic in
$T$
. We further show how to reduce the regret of a desired subset of the
$M$
costs, at the expense of increasing the regrets of rewards and the remaining costs. To the best of our knowledge, ours is the only work that considers nonepisodic RL under average cost constraints and derives algorithms that can
tune the regret vector
according to the agent's requirements on its cost regrets.