Augmented Memory Replay in Reinforcement Learning With Continuous Control

Online reinforcement learning agents are currently able to process an increasing amount of data by converting it into a higher order value functions. This expansion of the information collected from the environment increases the agent’s state space enabling it to scale up to more complex problems but also increases the risk of forgetting by learning on redundant or conflicting data. To improve the approximation of a large amount of data, a random mini-batch of the past experiences that are stored in the replay memory buffer is often replayed at each learning step. The proposed work takes inspiration from a biological mechanism which acts as a protective layer of higher cognitive functions found in mammalian brain: active memory consolidation mitigates the effect of forgetting previous memories by dynamically processing the new ones. Similar dynamics are implemented by the proposed augmented memory replay or
AMR
algorithm. The architecture of
AMR
, based on a simple artificial neural network is able to provide an
augmentation policy
which modifies each of the agents experiences by augmenting their relevance prior to storing them in the replay memory. The function approximator of
AMR
is evolved using genetic algorithm in order to obtain the specific
augmentation policy
function that yields the best performance of a learning agent in a specific environment given by its received cumulative reward. Experimental results show that an evolved
AMR
augmentation function capable of increasing the significance of the specific memories is able to further increase the stability and convergence speed of the learning algorithms dealing with the complexity of continuous action domains.