Understanding the Role of Convolutional Layers in Deep Neural Networks for EEG-Based Emotion Recognition

Emotion recognition allows estimation of users' emotions from biological signals such as facial expressions, voice, and brain signals. Emotion recognition based on the electroencephalogram (EEG) is considered to be particularly reliable because the user cannot intentionally hide his or her reactions. When using conventional machine learning methods for emotion recognition from EEG, it is necessary to apply preprocessing to input raw EEG data and to exploit emotion-specific characteristics and perform feature extraction appropriately. On the other hands, deep learning methods do not require manual feature extraction, and therefore research on emotion recognition from EEG using deep learning has been active. However, deep learning models and their classification process are often regarded a black box because it is difficult to interpret the decision-making process of the model. In this study, it was attempted to improve the interpretability of deep learning models by clarifying the functional role of convolutional layers in deep learning models for EEG-based emotion recognition. First, we applied a band-pass filter to the EEG to assess the relationship between the frequency band of the input EEG signal, and the accuracy of emotion recognition by using it as input to a deep learning model that combines a CNN and an LSTM. Next, to clarify the functional role of the convolutional layer in the operation of the model and to improve the interpretability of the deep learning model, the classification accuracy of the deep learning model without the convolutional layer was evaluated using the pre-processed EEG data using the time-frequency analysis as input. The results indicated that the convolutional layer contributes to the selection of frequency components.