Pattern Discovery and Anomaly Detection via Knowledge Graph

In this paper, we developed a pattern discovery and anomaly detection system using a knowledge graph constructed by integrating data from heterogeneous sources. Specifically, the knowledge graph is constructed based on data extracted from structured and unstructured sources. Besides the extracted entities and relations, the knowledge graph finds hidden relations via link prediction algorithms. Based on the constructed knowledge graph, the normalcy model for entity, action, and triplets are established. The information of the incoming streaming data is extracted and compared to the normalcy model in order to detect abnormal behaviors. In addition, we apply the lambda framework to enable a computationally scalable algorithm for pattern discovery and anomaly detection in a big data environment. Real time tweets data are used for evaluation and preliminary results show promising performance in detecting abnormal pattern and activities.