Stochastic real-valued reinforcement learning to solve a nonlinear control problem

This paper presents a new approach to reinforcement learning (RL) to solve a nonlinear control problem efficiently in which state and action spaces are continuous. We provide a hierarchical RL algorithm composed of local linear controllers and TD-learning, which are both very simple. The continuous state space is discretized into an array of coarse boxes, and each box has its own local linear controller for choosing primitive continuous actions. The higher-level of the hierarchy accumulates state-values using tables with one entry for each box. Each linear controller improves the local control policy by using an actor-critic method. The algorithm was applied to a simulation of a cart-pole swing-up problem, and feasible solutions are found in less time than those of conventional discrete RL methods.