Multi-layer Embedding Neural Architecture with External Memory for Large-Scale Text Categorization

Text categorization proves to be very effective for many language processing applications. Over the years, with the advent of different machine learning entities, the prediction of text categories becomes easier even for a very large document after subtle extraction of features from the documents. Particularly, the learning of sequential features from text data can achieve the state-of-the-art prediction performance. Traditional sequence based deep learning algorithms, namely recurrent neural network (RNN) and its variants, learn and embed features effectively to categorize documents in terms of sequences of a document. However, the memory capacity of these algorithms to track the sequences is limited and less accurate, particularly for the document with a large context. This paper studies an external memory based sequential learning framework to overcome the memory capacity problem. In this work, we propose to employ a multi-layer embedding neural architecture with external memory for the large-scale text categorization problem. It employs long-short-term memory (LSTM) layers as the controller to capture short-term sequential patterns longer and utilizes an external memory to memorize long-term contextual information. An embedded feature layer is introduced to generate one-hot feature vectors from the text. The whole neural architecture is fully differentiable and thus it can be trained in an end-to-end manner. Our experimental results show that the proposed text categorization method outperforms the state-of-the-art methods over a number of large-scale text categorization benchmarks.