Adaptive control of Markov chains, I: Finite parameter set

Consider a controlled Markov chain whose transition probabilities depend upon an unknovn parameter α taking values in finite set A. To each a is associated a prespecified stationary control law ϕ(α). The adaptive control lay selects at each time t the control action indicated by ϕ(αt) where αt is the maximum likelihood estimate of α. It is shown that αt converges to a parameter α* such that the 'closed loop transition probabilities corresponding to α* and ϕ(α*) are the same as those corresponding to α0 and α(#x003B1;*) where α0 is the true parameter. The situation vhen α0 does not belong to the model set A is briefly discussed.