Revealing the landscape: Detecting trends in a scientific corpus

Scientific literature is growing at a rapid pace. Meanwhile, various stakeholders need to grasp novel trends and innovations in order to make sound policy decisions in real time. Modern data mining techniques can be leveraged to ease the burden of manually combing through thousands of documents. In this paper we present a data product for exploration and filtration of a large corpus of citations from several distinct commercial databases. The end goal is a highly interactive user interface for intuitive corpus exploration on the front end, supported by data ingestion, merging, inference, and novelty detection capabilities on the back end. Due to the high dimensionality of textual data, dimensionality reduction and summarization are major requirements for effective exploratory analysis. Toward these ends we apply a topic model, specifically the Latent Dirichlet Allocation (LDA) model. The resulting dimensionality reduction improves the comprehensibility of the corpus for the end user, while also allowing a speedup of document-document comparison. Document similarity is computed using Hellinger distance, which is a Euclidean distance in a transformed topic-weight space, and thus nearest document queries can be implemented efficiently using a kd-tree data structure. Further information reduction is achieved through the use of a supervised nonparametric trend detection algorithm originally developed in the context of social media (Twitter), in order to suggest terms of potential interest to the user based on their likelihood of embodying significant trends. To our knowledge the application of this technique in the scientometric domain is novel.