Connecting Researchers and Grant Opportunities: A Deep Learning Approach to Extract Data from Heterogenous Unstructured Sources

Deep learning-based recommender systems are widely utilised in domains such as e-commerce. Yet there are limited studies that explore recommendation systems for expert and speciality needs such as finding grant opportunities or job vacancies in a specific field. One reason for this is the lack of large volume of homogenies data of good quality. The aim of our research is to build a data analytics pipeline for an explainable recommender system that can handle heterogenous data sources and imperfect data. The data sources of interest range from structured, semi-structured to unstructured data. We propose a novel domain knowledge-guided BERT based question and answering (Q&A) approach to extract relevant contextual information from multiple relevant sources of information. To verify the quality of the developed data pipeline, our pipeline has an embedded GenerativeAI model based statistical quality monitoring system. The interaction with the GenerativeAI model for quality checking is designed following the architecture of human-computation techniques by considering different prompt engineering strategies. We demonstrate the capabilities of the proposed method through a case study in the area of grant opportunity recommendation for the academic researchers.