Dual Memory Structure for Memory Augmented Neural Networks for Question-Answering Tasks

Memory is crucial for machine learning tasks on sequential data. From vanilla RNN to LSTM and memory augmented neural networks, researchers have investigated several types of memory structures. However, they suffer from limitations in the capacity or ability to keep track of long-term dependencies. This paper presents an external memory module composed of two distinct submodules that are inspired by memory in the human brain. Besides, a sleep mechanism is incorporated into this memory, which can mimic sleep’s effects on improving human memory. The proposed method is fully differentiable; thus, backpropagation can be used for its training. Experiments conducted on the bAbI dataset show that the proposed method is successful in 16 out of 20 tasks, and the average error is 2.8%. The performance of the proposed method is far better than the conventional NTM, and it has the lowest prediction error in 7 out of 20 tasks among baseline systems. Besides, the proposed system is the only system that can solve tasks 16 and 17 of the bAbI dataset.