The Rhythm of Flow: Detecting Facial Expressions of Flow Experiences Using CNNs

Recently, the flow state, a state in which individuals perform at the peak of their ability and are completely immersed in the task while experiencing a state of elatedness, has been the subject of active research. We introduce a novel approach of using convolutional neural networks to recognize flow in live performing musicians from analyzing their facial expression. A modified and partially re-trained version of the popular ResNet-50 architecture is employed for binary classification of flow, achieving a detection accuracy of 77.55 percent. This is done on labelled YouTube video-data of musicians with a labeling strategy that was verified through a perception experiment. Maximum accuracy within a 5-fold cross-validation is 74.98 percent with the mean exhibiting an accuracy of 65.10 percent. The results indicate that the state of flow is indeed recognizable through facial expressions of musicians. In addition, the utility of the presented model is demonstrated in two exemplary applications: Predicting the popularity of YouTube videos based on flow recognized in the faces through our system and correlating flow and six discrete emotions (neutral, happy, angry, fear, disgust, surprise).