Weighted-Norm Bounds on Model Approximation in MDPs with Unbounded Per-Step Cost

We consider the problem of designing a control policy for an infinite-horizon discounted cost Markov Decision Process (MDP)
$\mathcal{M}$
when we only have access to an approximate model
$\hat{\mathcal{M}}$
. If we design an optimal policy
$\hat{\pi}^{\star}$
for the approximate model, how well does it perform when used in the true model
$\mathcal{M}$
? We provide an answer to this question by bounding a weighted norm of the difference between the value function of
$\hat{\pi}^{\star}$
when used in
$\mathcal{M}$
and the optimal value function of
$\mathcal{M}$
. The use of weighted norm allows us to obtain meaningful bounds for performance loss even when the per-step cost function is unbounded. This is in contrast to much of the prior literature which has largely focused only on the case of bounded per-step cost. We illustrate our results for two specific instances - a finite MDP model for an inventory control problem and the discounted linear quadratic regulator problem.