Controlled Markov chains with discounted risk-sensitive criteria: Applications to machine replacement

We analyze controlled Markov chains (CMC) with a risk-sensitive exponential discounted cost criterion. We present dynamic programming results for the finite horizon problem. We derive an optimality equation for the infinite-horizon case from which the optimal and exponential discounted cost and a utility-optimal policy can be obtained. A value iteration result is also given. These results were previously obtained by Chung and Sobel (1987) and others for a finite state space for the optimization of the set of deterministic Markovian policies. We show that those results are valid for an infinite countable state space and we consider the set of all admissible policies. We apply the results of the previous section to study the consequences of introducing risk sensitivity in an equipment replacement problem. Our interest was kindled by Fernandez-Gaucherand and Marcus (1994) who study risk-sensitivity in a finite horizon equipment replacement problem with partial state information. Finally, we show that there exists an optimal threshold ultimately stationary policy and conclude this paper by showing that the tail of that policy is a stationary risk-neutral optimal policy.