Discrete versions of an algorithm due to Varaiya

In this correspondence, we present a discrete formulation of an algorithm proposed by Varaiya for the expected average cost per unit time Markov decision problem, a method for selecting an important parameter, some techniques for improving the discrete algorithm, and results of a numerical comparison with other algorithms.