A novel design for a gated recurrent network with attentional memories

Recurrent neural networks (RNN) are used for pro-cessing the temporal dynamics of complex sequential information. Existing recurrent neural networks, such as Long Short-Term Memory (LSTM) and its variant, Gated Recurrent Unit (GRU), are used to control current and historical information. This study proposes a novel design for a gated recurrent neural network with an attention mechanism, named the Attentional Memory Unit (AMU), which gives the recurrent neural network an attention capability. This design uses a gated mechanism and a generalized attention mechanism so that it can be adaptively adjusted in the spatial and temporal sense to allow the application of the attention mechanism for different scenarios. The experimental results verify that the proposed AMU significantly boosts the power of RNNs.