Approximate policy improvement for continuous action set-point regulation problems

Model-free control methods based on classical dynamic programming approximation algorithms are a fertile area of research. These methods are generally for discrete state and action spaces. This research proposes an algorithm for extending approximate policy iteration to continuous action spaces by combining the derivative-free line search methods of nonlinear optimization with policy improvement based on Q-learning using a discrete subset of reference actions. The properties of the proposed algorithm are discussed and two example problems illustrate its applicability. Q-learning algorithms are used to search a continuous action space. The method reduces the computational effort required in many problems.