Multimodal Decision Fusion of EEG and fNIRS Signals

Fusion of electroencephalography (EEG) as a physiological signal and functional near-infrared spectroscopy (fNIRS) as a metabolic signal has enormous potentials in different applications of biomedical engineering. A decision fusion model of a multimodal system combines the output decisions or predictions of each unimodal classifier. The proposed study exploits a long short-term memory network (LSTM) classifier which classifies EEG and two parts of a fNIRS signal: oxyhemoglobin (HbO
2
) and deoxyhemoglobin (HbR) separately in 3 different classes of the cognitive tasks (ternary classification). The output decisions of LSTM classifiers are combined in a support vector machine (SVM) classifier. A receiver operating characteristic (ROC) curve measures performance of the proposed model based on the fused decisions of the SVM meta classifier. The ROC and different performance evaluation parameters show that the proposed EEG-fNIRS decision fusion model based on LSTM and SVM classifiers outperforms any single modality. The improvement in detection rate for 26 subjects in decision-fused modality is +31.83%,+5.2%, +15.19% compared to EEG, HbO
2
and HbR alone respectively.