Nonlinear Value Function Approximation Method With Easy Hyperparameter Tuning and Convergence Guarantee

Reinforcement Learning is a branch of machine learning to learn control strategies that achieve a given objective through trial-and-error in the environment. Because this can be applicable even when the state transition function of the control object is unknown or it is difficult to create its model properly, this can reduce the designer’s burden. Reinforcement learning repeats evaluating a policy and improving it based on that evaluation. An algorithm called value function approximation is used in this evaluation. Value function approximation is a general term for algorithms of approximating a function called value function, which maps an arbitrary state to how much reward can be obtained in the future when acting according to a policy from that state. Value function approximation should have high approximation accuracy and hyperparameter easy to tune. Although various value function approximation methods have been proposed so far, some of them does not have convergent guarantee and the others’ hyperparameter tuning is difficult. The purpose of this paper is to propose a value function approximation method that have convergence guarantee even with nonlinear function approximator and whose hyperparameter tuning is easy and to verify its performance through numerical experiments. To this end, we focused on GTD2 method, which is one of the value function approximation methods that has convergence guarantee even with a nonlinear function approximator. GTD2 method has the drawback that its hyperparameter is difficult to tune appropriately so that the approximation does not diverge during learning. Therefore, we firstly clarified the cause of the divergence of GTD2 method. Secondly, based on this cause, we proposed Normalized and Regularized GTD2 (NRGTD2) method, that incorporates a method for suppressing divergence into GTD2 method. Finally, through numerical experiments, it was clarified that the proposed method suppresses the divergence and can achieve better approximation accuracy than conventional methods by easy hyperparameter tuning.