A Dual-Staged Attention Based Conversion-Gated Long Short Term Memory for Multivariable Time Series Prediction

In multivariate time series modeling, it is necessary to capture short-term mutation and long-term dependence information simultaneously. However, mechanism which can capture short-term change is difficult to be used to grasp long-term dependence information, and vice versa. In order to capture both short-term mutation and long-term dependence information in the same model, this paper proposed a dual-staged attention mechanism based on conversion-gated Long Short Term Memory network(DA-CG-LSTM). Hyperbolic tangent function is introduced into the input-gate and the forget-gate of Long Short Term Memory network(LSTM), which improves the ability of the network to extract the short-term mutation information. Further, dual-staged attention mechanism is added in the network, which includes input attention and temporal attention. Input attention adaptively extracts the feature relations of exogenous sequences, and temporal attention selects the relevant hidden layer states across all the time steps. Experiments on air quality and traffic flow time series data show that the proposed network has lower average absolute error, average absolute percentage error and root mean square error by more than 50% compared with Dual-staged Attention Recurrent Neural Network(DA-RNN) and Transformation-gated LSTM(TG-LSTM).