A User-Centric Evaluation of a Continuing Education Course Recommender System

The global labor market is changing due to artificial intelligence (AI), causing job losses and new opportunities. One predicted consequence is the emergence of a discrepancy between the skills possessed by employers and employees and those offered by continuing education providers. AI-based systems, such as recommender systems (RS), could help bridge this gap. RS are commonly used in educational settings, though their application in continuing vocational education (CVE) remains limited. This study evaluates two deployed hybrid explainable RS versions for CVE courses provided by a German startup-one using knowledge graphs combined with collaborative filtering and the other employing a deep neural network. These RS generate recommendations from a database of 1,500 courses that six German CVE course providers deliver. The goals of this evaluation study are twofold: to evaluate the status quo of the RS and to explore potential improvements. We evaluated this RS by conducting 26 semi-structured interviews with neutral CVE course counselors in two settings. Group one worked with the RS on fictional cases, while group two used it for a month in real situations. Participants saw value in the RS but noted that the quality of explanations was insufficient. Both groups found the RS easy to use but suggested improvements, such as adding questions on preferred learning formats to enhance recommendations.