Unpacking the CNN’s Algorithmic “Glass Ceiling” of Inferring Psychological Traits from Facial Images

Deep learning techniques have brought novel analytic tools into multiple scientific fields. As one of the traditional domains in social psychology, face perception, which investigates the association between facial physical features and inferred psychological traits, has begun to benefit from deep learning techniques either, specifically for Convolutional Neural networks (CNNs). However, CNN’s predicting performances are not identical among various traits, even when utilizing the same CNN algorithm. Several researchers pointed out that it is due to the variations of inter-rater agreements on different traits. Besides this, we suggest that the variability of the trait ratings in the same image databases (i.e., standard deviation of trait evaluation) and the belonging clarity degree of a trait positioning in the underlying structure of many traits (i.e., absolute maximum load) could also explain the performance variation of CNN algorithm in trait judgment predictions. Drawing on the 10k US Adult Faces Database and the results of 40 social traits obtained by Song, A. et al. (2017) using the CNN algorithm, we employed inter-rater agreement, standard deviation of evaluation, and absolute maximum load in a regression equation to predict the variation of the fitting degree between CNN’s predicting evaluations and human’s ratings (goodness of fit of CNN). The result showed that inter-rater agreement explains a large part of the variance, as well as the standard deviation of evaluation and absolute maximum load, could independently account for a substantial part of the variance. Then, we utilized the structural equation model (SEM) to further lay out the above predictors’ pathway effects on the CNN’s goodness of fit. The finding has important implications for us to understand how deep neural networks learn human face perception and the prediction accuracy of CNN on human face perception judgments could be increased.