Improving Word Representations with Document Labels

Word representations, aiming to build vectors for each word, have been successfully used in a variety of applications. Most word representations are learned from large amounts of documents ignoring other information. This is rather suboptimal because the side information of the documents, such as document labels, is not used in learning word representations. In this paper, we focus on how to exploit these side information to improve word representations. We propose to incorporate document labels into the learning process of word representations in two frameworks: neural network and matrix factorization. The experimental results on word analogy and word similarity task show that our models can better capture the semantic and syntactic information than the original models. Our models also improve the performance of word representations on text classification task.