Toward an Interactive Reading Experience: Deep Learning Insights and Visual Narratives of Engagement and Emotion

Engagement and emotion are critical components that significantly influence a reader’s experience during a reading task. Despite the crucial role of engagement and emotions in shaping our reading experience, accurately tracking these dynamic states during actual reading remains a significant challenge. This study bridges this gap by detecting engagement and emotion levels during a reading task by leveraging the power of state-of-the-art deep learning models and investigating the correlations between the engagement levels and emotions. An experiment was conducted involving 18 university students reading 14 documents followed by a questionnaire to rate their levels of engagement, valence, and arousal after reading each document. A Tobii 4C eye-tracker with a pro license along with an Empatica E4 wristband were utilized to record behavioral and physiological data from the participants. A range of deep learning models were utilized for computing the engagement, valence, and arousal values, employing both user-independent and user-dependent methods. Our investigation revealed distinct yet complementary strengths in two deep learning models: Transformer excelled in user-independent detection of engagement and emotion with an accuracy of 80.38% (engagement), 71.28% (arousal) and 73.98% (valence) while ResNet shined in the user-dependent setting with an accuracy of 93.56% (engagement), 90.62% (arousal) and 88.70% (valence) which highlights the interplay between individual differences and reading dynamics. Intriguingly, we observed strong, document-specific correlations between engagement and emotion states, suggesting that different texts evoke unique affective responses. We developed an interactive dashboard visualizing predicted engagement and emotions, offering real-time feedback and personalized learning possibilities. The dashboard features an engagement gauge that displays the reader’s level of engagement based on predicted class probabilities, and an emotion emoji serving as a visual cue that illustrates the predicted emotional state of the reader. This technology can inform the design of dynamic interfaces that adjust to individual reading styles and emotional responses, potentially enhancing comprehension and involvement.