Usability and User Experience Related Perceptions of University Students Regarding the Use of Bing Chat Search Engine and AI Chatbot: Preliminary Evaluation of Assessment Scales

This paper presents a preliminary investigation of usability and user experience characteristics of Bing Chat, a GPT based artificial intelligence (AI) chatbot that is integrated with the Microsoft Bing search engine. The advances in conversational AI with services like GPT-4 and Bing Chat have created considerable challenges and opportunities for academic education. With a focus on the potential positive aspects of the Bing Chat use, selected variables that are broadly related to usability and user experience of students were investigated in a pilot study. A survey was applied on two convenience samples of students from different Croatian higher education institutions (N=30 and N=43). The survey included assessment scales for the following variables: Perceived Usefulness, General Usability, Learnability, System Reliability, Visual Design and Navigation, Information Quality, Information Display, Cognitive Involvement, Design Appeal, Trust, Risk Perception, and Intention to Use. For most of the assessment scales an acceptable Cronbach alpha coefficient as an indicator of internal consistency was revealed. The following variables had the highest correlation with Intention to Use Bing Chat (or similar service like GPT-4): Cognitive Involvement, Design Appeal, and Trust. Students' responses to specific survey items revealed an overall positive evaluation of various usability and user experience characteristics of Bing Chat. The results of our preliminary study are discussed in the context of technology adoption regarding Bing Chat, as well as the design of learning activities for students in higher education.