Partially Recurrent Network With Highway Connections

It is difficult to train deep recurrent neural networks (RNNs) to learn the complex dependencies in sequential data, because of gradient problems and learning conflicts. To address these learning problems, this article proposes a partially recurrent network (PR-Net), which consists of the partially recurrent (PR) layers with highway connections. The proposed PR layer explicitly arranges the neurons into a memory module and an output module, which are implemented by highway networks. The memory module places memory on the information highway, to maintain long short-term memory (LSTM). The output module places external input on the information highway, to approximate the complex transition function from one step to the next. Because of the highway connections in the output module, the PR layer can pass gradient information across layers easily, to allow the training of the PR-Net with multiple stacked PR layers. Furthermore, the explicit separation of the memory module and output module can reduce the learning burden of RNNs that which neurons are responsible for memory and which are for output. With a comparable number of parameters, sublayers in the recurrent layer, and stacked recurrent layers, the proposed PR-Net outperformed the recurrent highway network and LSTM on three sequence learning benchmarks, significantly.

I. Introduction

Recently, the deep neural network (DNN) has achieved impressive results in various applications, such as computer vision [1]–[6] and natural language processing [7], [8]. As a class of DNNs, recurrent neural networks (RNNs) are widely used in the field of sequence learning [7]–[10]. It is a class of biologically inspired artificial neural networks that model the feedback connections by recurrent connections [11]. Because of the recurrent connections, RNNs can maintain the working memory (activations of neurons), and merge the memory of the history with the current information to produce predictions [12].