Multimodal stability-sensitive emotion recognition based on brainwave and physiological signals

This paper presents a framework for adaptive multimodal emotion recognition based on signal stability as a context. To verify the efficacy of the method, experiments were conducted using a dataset of brainwave and physiological signals (EEG, ECG, GSR) captured from nine subjects listening to music. The proposed method uses a combination of signal-based features as well as accelerometer data to quantify the approximate reliability of each modality. In contrast to existing approaches, unstable modalities are not rejected outright, instead their relative contribution is dynamically adapted based on a corresponding stability index. In the case of EEG, the stability index was calculated using an artifact rejection technique, while for the ECG and GSR modalities it was calculated based on body movement detected through accelerometers. The experimental results show that temporally varying the relative contribution of each modality can improve emotion recognition performance.