Multimodal detecting and analytic system for visual perception and emotional response

The correlation between visual perception and the evoked emotional response has been investigated extensively in the area of cognitive psychology using the method of electroencephalography (EEG). Activities in the visual cortex have been found to be correlated with psychophysical performance, visual attention, and subjective perceptual experience. While electroencephalography could record the cerebral activities accurately, the subject's visual attention was always left out of consideration. In the study, we proposed and developed a multimodal detecting and analytic system for visual perception and emotional response using a combination of electroencephalography, eye tracking and heart rate. We maximized the portability of the system while ensuring the accuracy and reliability of the data. The effectiveness of the system was tested by an experiment based on negative, positive and neutral images selected from ThuPIS image system. 10 participants took part in the experiment and their emotions were analyzed in valence and arousal dimensions as they were observing the images. The results indicated positive correlations between valence and power of electroencephalography signals over the occipital lobe at low frequencies and over the left and right temporal lobes at high frequencies. The saccade magnitude and fixation duration of eye movements could also function as indicators of emotion. We designed a compact module of the system for psychological diagnosis and treatment. Our system may contribute to improve the precision of visual-based psychological treatment by providing real-time emotion analysis.