Risks and Benefits of AI-generated Text Summarization for Expert Level Content in Graduate Health Informatics

AI-generated text summarization (AI-GTS) is now a popular topic in applied computer science education. It has proven helpful in various sectors, but its benefits and risks in education have not been thoroughly investigated. Few researchers have demonstrated the benefits of employing AI-generated text summaries in learning to generate ideas swiftly and to explore insights and hidden knowledge. AI-GTS has made it easier for students to understand electronically-available critical information. On the other hand, the risks linked with its implementation in education are understudied. Some anticipated risks include harming pupils' writing skills, overdependence, reduced critical thinking capacity, and increased plagiarism. This paper presents the application of AI-generated text summarization in a graduate health informatics course and discusses the risks and benefits to students. Furthermore, utilizing the Bidirectional Encoder Representations from Transformers (BERT) model, we demonstrate that the current state-of-the-art AI-generated text summarization has the potential to create expert knowledge content. We conducted a study with 58 health informatics graduate students in the Fall of 2019 to write annotated bibliography for 25 articles each, to which we also added the AI-generated article summaries. We then asked the students to peer grade and distinguish the AI-generated annotations from the student-written summary. Using the Kruskal-Wallis test, we found no significant difference in the peer grades between the two. The robustness of such AI-generated text summarization raises important questions for educators teaching in health informatics.