Emotional Estimation Method through Fusion of Temporal Skin Temperature and Eye State Changes

Non-contact facial expression recognition has increased owing to technological advancements and the COVID-19 influence, which is vital for assessing digital content interest and quality. Studies demonstrate that changes in facial skin temperature, particularly in the nose and cheek areas, can be used for emotion estimation, with thermal images outperforming visible images. However, research on time-series data and the integration of thermal and visible images remains limited. In this study, we propose a method for estimating strong emotional arousal during digital content viewing using time-series data from thermal and visible images. Consequently, we achieved an estimation accuracy of over 80% by combining changes in facial skin temperature and the eyes opening degree.

I. Introduction

Recently, non-contact facial expression recognition using skin temperature has become vital for assessing digital content interest due to technological advancements and COVID-19. This technology is crucial for measuring viewer engagement and content quality. Previous studies have demonstrated changes in facial skin temperature, particularly in the nose and cheeks, during emotional arousal [1]. Additionally, thermal images are more effective than visible images for emotion estimation [2]. However, the integration of time-series data from thermal and visible images remains underexplored. Existing studies often emphasize static analyses or single-modality approaches, failing to capture dynamic emotional responses.