Improved Stochastic Synapse Reinforcement Learning for Continuous Actions in Sharply Changing Environments

Reinforcement learning in continuous action spaces requires mechanisms that allow for exploration of infinite possible actions. One challenging issue in such systems is the amount of exploration appropriate during learning. This issue is complicated further in sharply changing dynamic environments. Reinforcement learning in artificial neural networks with multiparameter distributions can address all aspects of these issues. However, which equations are most appropriate for updating these parameters remains an open question. Here we consider possible equations derived from two sources: The classic equations proposed for REINFORCE and modern equations introduced for Stochastic Synapse Reinforcement Learning (SSRL), as well as combinations thereof and variations thereon. Using a set of multidimensional robot inverse kinematics problems, we find that novel combinations of these equations outperform either set of equations alone in terms of both learning rate and consistency.