Constructing a Closed-Domain Question Answering System with Generative Language Models

Generative language models such as ChatGPT are good at producing answers for questions whose answers are publicly available. For questions about private organization documents, such models cannot perform well due to the fact that they were not trained on these documents, hence the knowledge was not embedded in their parameters. On the other hand, traditional Question Answering (QA) systems with a Retriever and a Reader model are interpretable and can be trained quickly on private documents, but they require laborious annotation. The output of QA systems is a span of text, which is not friendly to the end users as well. In this paper, we proposed a framework for generating closed-domain QA data set in a semi-automatic manner, reducing human efforts. An organization-specific QA data set was created based on this framework. Additionally, we fine-tuned a traditional open-domain QA model on the newly created data set, and combine the output of this model with recently released language models to improve the naturalness of the response. Finally, we present our results and discuss the findings.