CEMDQN: Cognitive-inspired Episodic Memory in Deep Q-networks

Reinforcement learning in the field of artificial intelligence has seen tremendous advances in recent years, but there are still several limitations standing in the way of its wider practical application, including sample inefficiency, generalization, and exploration-exploitation trade-off. Deep Q-Networks (DQN) have improved the performance of RL by using deep neural networks to approximate the Q-function and by using experience replay to store and reuse past experiences. Episodic memory in RL is a technique that allows an agent to store and reuse past experiences in order to improve its decision-making. However, current episodic memory-based RL techniques have some issues, such as generalization and slow learning, which can be improved by using methods such as experience replay compression and reducing the information of episodic memory into a parametric model. In this work, we propose cognitive-inspired episodic memory in DQN networks (CEMDQN) that reduces the priority weighting of old experiences over time, and the agent accesses recent experiences more frequently. The proposed model was evaluated on three different environments: StarGunner, BattleZone, and TimePilot. It was shown that when compared to standard episodic memory DQN, CEMDQN was more effective in test score performance for StarGunner (45.1 %), BattleZone (81 %), and TimePilot (53.54%) environments, respectively.