A Scalable Automated System to Measure User Experience on Smart Devices

In this paper, we present an automated scalable system that measures user experience on smart devices such as TVs, tablets and smart phones. The system consists of three parts: (i) a robot with a mobile arm to perform touches and clicks on a tested device such as a tablet or a phone, and sensors to capture the video signals, (ii) a signal capturing process records the input video in real time, controlled by algorithms that use a deep detection model and a text matching model to estimate app state, and a deep classification model to navigate, and (iii) a quality metrics computation process that uses spatial and temporal computer vision algorithms. We show that this system can continuously evaluate major streaming apps such as the Prime Video app, popular shopping apps such as Amazon retail app, and other mobile apps. We also do manual validation for a subset of the metrics to evaluate the reliability of our system.