Recognition of Emotional Categories Using Mined Fuzzy Rules From Electroencephalogram Signals With Gated Eye Track Data Approach

Researchers suggest that a short video clip, when tagged with a single label, is sufficient for classification into an emotional category. However, when subjects view an emotional film tagged similarly, there is no guarantee that the designated emotion persists throughout the duration, or when exactly the emotion is elicited. This inconsistency adversely affects the performance of emotion recognition (ER) systems. In this study, we propose a multimodal ER system employing an eye-tracking gated strategy to identify the most effective timing for emotional categorization. Initially, common eye-tracking features are extracted and selected using the minimum-redundancy-maximum-relevance (mRMR) method. Subsequently, the most discriminative feature is employed as a threshold to pinpoint the most relevant timing. EEG signals from these moments are then decomposed into five standard frequency bands using the Daubechies wavelet function (order 4). Furthermore, four types of entropy features are extracted from four-second segments of 62 and 32 channels for the SEED-IV and MAHNOB-HCI databases, respectively. The best features, as determined by the mRMR method, are fed into a Sugeno-fuzzy inference system (S-FIS) designed to derive rules for discriminating between the four emotional categories of happiness, fear, sadness, and neutrality. The S-FIS rules were refined using a genetic algorithm (GA), leading to most discriminative rules achieving average accuracies of 94.42% and 85.20% for the alpha frequency bands of the SEED-IV and MAHNOB-HCI databases, respectively. The results of this study demonstrate the effectiveness of fuzzy rule extraction in enhancing the performance of multimodal ER systems