Pre-Trained Generative Architectures for Question-Asking Chatbots on Technical Text: A Case Study

In recent years, chatbots' usefulness has spread to an ever-expanding number of domains. Chatbots can be used in educational contexts to ask students questions about topics they are studying for self-testing. Designing models for dialogue systems includes the crucial step of generating appropriate responses. In this study, we propose using chatbots which ask questions specifically catered to a technical domain of interest. To that purpose, we've chosen the fundamental concepts of the Java programming language as our domain. We begin by creating a collection of Java-based dialogues with various sets of question-answer pairings of varying difficulty across various topics. Our Java dialogue dataset includes 10000 multi-turn dialogues generated by an algorithm that simulates a Java interview on basic concepts. Every conversation serves as a training example for our proposed framework. Then, for training on our dataset, we use versions of two different transformer-based pre-trained generative architectures. We compare and evaluate the architectures based on standard automatic metrics for dialogue response generation and other parameters specific to our use-case. We observe that the usage of generative architectures pre-trained as dialog models can generate newer questions, in most cases meaningful in the context of the interview. However, the model design could benefit from incorporating a further knowledge component in order to improve the production of questions both in terms of topic and suitable level of difficulty.