Beyond the Game: Multimodal Game-Experience Recognition During Dynamic Affective Game Environments

The recent advent of Human-Computer Interaction (HCI), has highlighted the need of understanding human responses through multimodal signals during daily interactive experiences, such as serious games (SGs). In this study, we explore game experience responses (GER) recognition by analyzing the second Study in Bio-Reactions and Faces for Emotion-based Personalization for AI Systems (BIRAFFE-2) dataset, including multimodal data from 76 participants engaged in dynamic gameplay. By extracting features derived from electrocardiogram (ECG), electrodermal activity (EDA), accelerometer, gyroscope, game logs (GL), affect dynamics and personality traits (PT) fed in different machine learning models, our study focuses on GER prediction, achieving state-of-the-art performance metrics across different difficulty levels of SGs environments (accuracy: 0.951 for Tension in Optimal Game using Support Vector Machines). This highlights the importance of player’s game experiences as indicators for personalized HCI. Our approach sheds light on the intricate relationships between multimodal physiological signals, GL, users’ emotional backgrounds and personalities, contributing novel insights for the development of real-time adaptive and engagement-focused SGs. Notably, our study highlights the importance of ECG-Derived Respiration features and the influence of pre-game emotional states on GER.