Dynamic programming approach to a final-value control system with a random variable having an unknown distribution function

As described in the introduction of this paper, the state vector x
n
, representing the present state of a sampling control system, is assumed to satisfy a difference equation
x_{n+1} = T(x_{n}, r_{n}, \upsilon_{n})
, where
\upsilon_{n}
is the control vector of the system subject to random disturbances r
n
. The random variables r
n
are assumed to be independent of each other and to be defined in a parameter space in the following manner: Nature is assumed to be in one of a finite number,
q
, of possible states and each state has its own parameter value, thus specifying uniquely and unequivocally the distribution function of r
n
. It is further assumed that we are given the a priori probability
z =(z_{1}, . . . , z_{q})
of each possible state of nature,
i= 1,2, . . . , q,\sum\min{i=1}\max{q} z_{i}=1, z_{i} \geq 0
. Given a criterion of performance, the duration of the process and the domain of the control variable, a sequence of control variables {
\upsilon_{n}
} is to be determined as a function of the state vector of the system and time, so as to optimize the performance. The sequential nature of the determination of
\upsilon_{n}
is evident here, since the stochastic nature of the problem prevents specification of such a sequence of
\upsilon
's as a function of the initial state and time. By means ot the functional equation technique of dynamic programming, a recurrence relation of the criterion function of the process,
k_{n}(x, z)
, is derived, where
x
is the state variable of the system when there remain
n
control stages. When
q
is taken to be two and the criterion of performance to he
x_{N}^{2}
with the constraint on the control variables
\sum\min{i=0}\max{N-1} \upsilon_{i}^{2} \leq K
, where x
n
is the final state vector of the system, it is shown that
k_{n}(x, z)
is the form
w_{n}(z)x^{2}
and the optimal
\upsilon_{n}(x, z)
is linear in
x
, as might be expected. The dependence of
k_{n}(x, z)
and
\upsilon_{n}(x, z)
on
z
is investigated further, and
w_{n}(z)
is found to he concave in
z
. Explicit quadratic forms for
w_{n}(z)
are obtained in the neighborhood of
z = 0
and 1. The optimal
\upsilon_{n}(x, z)
is found to be monotonically decreasing in
z
. When the domain of
\upsilon_{n}
is restricted to a finite set of values, as in contactor servo systems, no explicit expressions for
k_{n}(x, z)
and
\upsilon_{n}(x, z)
are available. However,
k_{n}(x, z)
is still concave in
z
and approximately given by
zk_{n}(x, 1) + (1 - z)k_{n}(x, 0)
. By solving the recurrence relation numerically, this approximation is found to be very good for moderately large n, say 10. This means that if one has explicit solutions for
p =p_{1}
and
p = P_{2}
, then one can approximate the criterion function for the adaptive case where
Pr(p =p_{1})= z
by a linear function in
z
as shown above. This provides fairly good lower bound on
k_{n}(x, z)
, and in turn serves to determine an initial approximate policy. A concept of suboptimal policy is introduced and numerical experiments are performed to test suboptimal policy suggested from numerical solution. The system behavior under the suboptimal policy is also discussed.