Adaptive control of Markov chains, I: Finite parameter set

Consider a controlled Markov chain whose transition probabilities depend upon an unknown parameter α taking values in finite set
A
. To each α is associated a prespecified stationary control law
\phi(\alpha)
. The adaptive control law selects at each time
t
the control action indicated by
\phi(\alpha_{t})
where α
t
is the maximum likelihood estimate of α. It is shown that α
t
converges to a parameter α
^*
such that the "closed-loop" transition probabilities corresponding to α
^*
and
\phi(\alpha^{\ast})
are the same as those corresponding to α
⁰
and
\phi(\alpha)
where α
⁰
is the true parameter. The situation when α
⁰
does not belong to the model set
A
is briefly discussed.