Improving Confidence of Uncertain Knowledge Graphs by Crowdsourcing with Limited Budget

Knowledge graphs (KGs), either constructed automatically from texts or collected manually from crowdsourcing workers, may contain uncertainty. The uncertainty may propagate into the knowledge graph embedding and downstream tasks, which is potentially harmful, especially for those confidencesensitive applications such as medical diagnostic suggestion. Crowdsourcing workers with domain knowledge can help improve the data quality of knowledge graphs, by knowledge checking. However, due to the large scale of knowledge graphs and the limitation of adequate crowdsourcing workers, it is unrealistic to check all triplets in a knowledge graph to improve the data quality. Therefore, in this paper, we propose a crowdsourcing framework that efficiently improves the confidence of knowledge graphs with limited budget. We instantiate the framework in the medical domain and conduct a series of experiments with realworld medical data. We deploy the framework for knowledge graph embedding UKGE and corresponding downstream tasks. The experimental results show that the proposed method efficiently improves the quality of the knowledge graphs, and hence improves the performance of probabilistic knowledge graph embedding in the downstream tasks.