Usability Problem Identification Based on Explainable Neural Network in Asynchronous Testing Environment

Usability testing is an essential element in human–computer interaction studies, and its core purpose is not only to evaluate usability in general but also to identify specific problems. Synchronous usability testing methods are costly to conduct on a large scale, as well as subject to time constraints. Thus, in realistic practice, asynchronous testing is often performed and usually adopts an automatic logging approach to identify usability problems based on collected data. However, in prior research, an in-depth human evaluation review phase has been required to examine each user interaction for usability problems. This consumes considerable time, and the results are likely to be subjective. In this study, we propose a novel method to identify and prioritize usability problems quantitatively using an explainable neural network approach and conduct an experiment based on device usage logs collected in a usability test. Using an explainable neural network, we assess user interactions for their relative influence on an overall usability score, predicting each interaction’s probability of causing a usability problem. Assigning usability scores to each task, we model the relative importance of each individual interaction by calculating a weight for each based on feature maps output by the neural network. Then, we identify usability problems by reviewing the interactions showing the highest importance weight values. The experimental results indicate that our proposed method effectively identifies usability problems and demonstrate that it performs quantitatively better compared to other benchmark methods.