Evaluating generative artificial intelligence in answering course-related open questions: A pilot study

ChatGPT, which has been discussed in public in recent years, was launched in November 2022, with a continuously expanding market size. Generative Artificial Intelligence (GAI) has been developing rapidly in China. However, we have little knowledge of how well the GAIs are performing at answering course-related open questions, especially in Chinese. Therefore, this study aims to analyze the performances of GAIs in answering course-related open questions from both machine and manual evaluation perspectives. The results from this study showed that GAIs got unsatisfactory scores in this situation. Two different aspects of comparison of GAIs’ scores are discussed. More study on evaluating the performance of GAI is needed to investigate the capabilities of answering course-related open questions in the future.