On partially observable Markov decision processes with an average cost criterion

Consideration is given to partially observable Markov decision processes with finite or countable (core) state and observation spaces and finite control space. Following a standard approach, an equivalent completely observed problem is formulated, with the same finite control space but with an uncountable state space, namely, the space of probability distributions of the original core state space. It is observed that some characteristics induced in the original problem due to the finiteness, or countability, of the spaces involved are retained by the equivalent problem. Sufficient conditions are derived for a bounded solution to the average-cost optimality equation to exist. These results are illustrated in the context of machine replacement problems. Structural properties for average-cost policies are obtained for a two-state replacement problem and are similar to available results for discount optimal policies. The set of assumptions used seems to be significantly less restrictive than others currently available.<>