Optimization of stochastic finite state systems

A class of stochastic dynamic systems is considered in which the set
S
of allowable states, the set
Q
of all the inputs, and the set
O
of the outputs are all finite. For the subclass of the systems in which the state can be exactly measured, a method is given to find the optimal control, so as to optimize a suitable criterion function. The set of probabilities
Prob(q(t) = q_{k}/s(t) =s_{j}), q_{k} \in Q, s_{j} \in S
, where
q(t)
and
s(t)
are the input and state at time
t
, respectively, plays the role of control. The determination of the optimal control involves only a solution of a set of
N
difference equations, where
N
is the total number of states. These results will be extended for systems in which the measured output is noisy. In this case, by control one means the set of probabilities Prob
(q(t) = q_{k}/y(1),... , Y(t)), q_{k} \in Q
where
y(t)
is the measured output at time
t
. These probabilities are found to be products of the current estimate of the state of the system
s(t)
, based on all the available measurements with certain precomputable constants. These results are applied to the analysis of time-sharing computer systems like project MAC, and demonstrate how to choose an optimal queue discipline among the various available queue disciplines for scheduling the various users.