Observations on the practical use of adaptive critics

By studying adaptive critic designs (ACD) from the standpoint of practical use in training neural networks, we expect to establish the types of problems for which ACD might be preferable to more established methods. To restrict the scope, we have chosen to concentrate on applying ACD, specifically derivative critics, to the training of recurrent networks (L.A. FeldKamp et al., 1997). This is actually less restrictive than it may appear; many problems, including controller training, can be posed as optimizing some or all of the weights of a recurrent network. An immediate benefit of this focus has been to clarify the relationship between the derivatives that result from backpropagation through time (BPTT) and the quantities that derivative critics are expected to deliver. At the same time, many questions have been raised, such as that of the critic representation that best balances accuracy against the number of time steps required for adaptation. Because our formulation permits BPTT and derivative critics to be used together or separately, we expect that experience with a variety of problems will further clarify the various tradeoffs and suggest situations in which critics may be used to particular advantage.