Mining and Visualizing Associations of Concepts on a Large-Scale Unstructured Data

We investigate the problem of finding unknown associations between 'concepts' in a given text corpus. A 'concept' is an entity, which is referred to by a phrase or multiple phrases (in case an entity has several names or synonyms, e.g. "illness", "disease"), while an 'association' is a relationship defined in a particular domain. The pairwise associations computation poses major challenges when the size of concepts to be investigated is large. In this paper, we propose a new framework for computing and visualizing the association between concepts using the generic association measures and the public knowledge available in data sources such as Wikipedia. Core to our methodology are the pruning techniques employed to filter the pairs of concepts which are very unlikely to be associated. Moreover, we study the performance of different correlation measures in finding both 'direct' and 'indirect' associations between concepts. The indirect association analysis extends the utility of the proposed framework to a broader class of association mining applications including the interesting area of new hypothesis generation. The extracted associations in our framework are demonstrated using the Neo4J graph database, which not only provides a user-friendly visualization interface for observing the associations between concepts in a big graph, but also supports more advanced analytics such as 'community detection', 'centrality analysis', and SQL-like querying of the concept graph. We evaluate the performance of our proposed methodology on two datasets, including a 'gene-disease' association dataset from the DisGeNET public database, and 'disease-symptom-treatment' associations extracted manually from the "MedicineNet.net" website. Our results show that the proposed association measure is capable of finding 80% of true gene-disease associations with a false positive rate of 15%.