Model-free control of nonlinear stochastic systems in discrete time

Consider the problem of developing a controller for general (nonlinear and stochastic) discrete-time systems, where the equations governing the system are unknown. This paper presents an approach based on estimating a controller without building or assuming a model for the system. Such an approach has potential advantages in, e.g. accommodating systems with time varying dynamics. The controller is constructed through use of a function approximator (FA) such as a neural network or polynomial (no FA is used for the unmodeled system equations). This involves the estimation of the unknown parameters within the FA. However, since no functional form is being assumed for the system equations, the gradient of the loss function for use in standard optimization algorithms is not available. Therefore, this paper considers the use of a stochastic approximation algorithm that is based on a simultaneous perturbation gradient approximation, which requires only system measurements (not a system model). Related to this, a convergence result for stochastic approximation algorithms with time-varying objective functions is established. It is shown that this algorithm can greatly enhance the efficiency over more standard stochastic approximation algorithms based on finite-difference gradient approximations.