Who Is to Err? A Limited Effect of Data Labeling in Prediction of Users' Subjective Impressions of Web Designs

User Experience (UX) researchers and designers who seek to predict users’ subjective impressions nowadays turn to Machine Learning (ML) models trained on marked-up data. Labeling of graphical user interfaces (UIs) is a rather time-consuming process that is also prone to errors and inaccuracies, particularly if motivation of the human labelers is low. Correspondingly, the size of datasets in Human-Computer Interaction (HCI) rarely exceeds the order of 10,000, and the field largely relies on feature-based models. In our paper, we compare two principal ways of obtaining the features, with respect to their effect on the quality of the models that predict users’ impressions of web pages’ Complexity, Aesthetics and Orderliness. First, we calculated several layout-related metrics - balance, equilibrium, symmetry - for 368 website screenshots labeled by 40 human volunteers. Second, we obtained the similar metrics from ML-based online services working with raw web UI images. Our results suggest that despite the limitations of the current automated web UI parametrization tools, their metrics can actually yield superior models (having R²s higher by 107% on average). Our key implication for UI/UX designers and ML researchers is that the expected ROI should be carefully weighted before committing to the tedious manual web UI data labeling.