Emotion Recognition Based on Fusion of Local Cortical Activations and Dynamic Functional Networks Connectivity: An EEG Study

In this paper, we present a method to improve emotion recognition based on the fusion of local cortical activations and dynamic functional network patterns. We estimate the cortical activations using power spectral density (PSD) with the Burg autoregressive model. On the other hand, we estimate the functional connectivity networks by utilizing the phase locking value (PLV). The results of cortical activations and connectivity networks show different patterns across three emotions at all frequency bands. Similarly, the results of fusion significantly improve the classification rate in terms of accuracy, sensitivity, specificity and the area under the receiver operator characteristics curve (AROC), p < 0.05. The average improvement with fusion in all evaluation metrics are 6.84% and 4.1% when compared to PSD and PLV alone, respectively. The results clearly demonstrate the advantage of fusion of cortical activations with dynamic functional networks for developing human-computer interaction system in real-world applications.


I. INTRODUCTION
Emotion recognition facilitates the interaction between humans and intelligent machines.An emotion consists of complex mental activities that influence the physical and psychological behavior of a person during the process of social interactions and decision-making.Thus, emotion recognition is a critical factor for several domains such as human robot interaction, characterizing the level of interest on learning, identifying the level of vigilance in road and safety, detecting patient's mental and physical states and the progress in recovery [1]- [3].
Different approaches have been considered to measure emotions.These include the methods based on speech, facial expressions, physiological measurements and selfassessment [3]- [6].However, each of these poses certain limitations.For example, social expectations may bias self-assessment of emotions, speech, and facial expressions.

Subjects may conceal their feelings and hence influence these
The associate editor coordinating the review of this manuscript and approving it for publication was Alberto Botter.measures.Paralysis and skin diseases may also affect the facial expressions.In addition, emotion recognition through facial expressions require the subjects to stand in front of a camera to measure their expressions.
On the other hand, physiological signals measured from the autonomous nervous system (ANS) such as heart rate, respiration, and skin conductance yield a more objective measure of emotions than through facial expressions or voice [3], [7].However, these types of signals are sensitive to cardiovascular diseases, skin diseases, and physical activities [8], [9].In particular, various human activities can generate signals similar to those produced by ANS to emotional states, hence affecting the accuracy of emotion recognition.
To overcome the limitations mentioned above, researchers have utilized the physiological signals from the central nervous system (CNS) acquired using Electroencephalogram (EEG) technique.EEG signals provide a quantitative measure of the electric potentials generated from the brain in response to a particular stimulus.This has the advantages of being portable, low cost, easy to set-up, and having a high temporal resolution.
Researchers have focused on studying the relationship between emotional states and brain activities and have found a strong influence of cognitive processes on emotions [10]- [13].Several databases have been published in literature to evaluate different emotions.For example, ENTERFACE'06 [14] dataset with 54 EEG-channel recorded from 5 subjects is one such example.Another published database is the MAHNOB-HCI [15] database comprising 32 EEG-channel recorded from 27 subjects watching video clips with different emotions.Additionally, Koelstra et al. [16] published another public database called DEAP for the analysis of human emotions.This dataset contains 32 EEGchannel data of 32 subjects watching music video clips.Recently, SEED database [17], comprising 62 EEG-channel was recorded from 15 subjects while watching emotional video clips.Based on this dataset, researchers studied the correlation between EEG spectral power and valence scores in several studies.They found that higher frequency components on the frontal, parietal and occipital lobes had a higher association with the self-assessment based valence response.The same database [18] was used in another study in which the EEG power spectral density (PSD) and facial features were fused to improve the classification performance of continuous emotion recognition.Lin et al. [19] evaluated the emotion-specific features based on the PSD changes of EEG and assessed the association between EEG dynamics and music-induced emotional states.The study found that the features from frontal and parietal lobes provided discriminative information and were strongly associated with emotion processing.
In a similar way, other researchers have evaluated emotions (positive, neutral and negative) using EEG signals alone or in combination with other physiological signals including eye tracking data and facial expressions, and have achieved classification accuracy in the range of 58% to 80% [20]- [30].Although these studies have tried to classify different emotions by statistically analyzing EEG signals, most of them focused on EEG features extracted at the single-electrode level.On the other hand, Lee et.al.[31] classified three different emotions; neutral, positive and negative by means of functional connectivity.The authors highlighted the importance of functional connectivity in emotion detection and showed that the classification accuracy outperformed those using power spectral features at single electrode level.In line with this, Mauss and Robinson in their review paper, have indicated that ''emotional state is likely to involve circuits rather than any brain region considered in isolation'' [3].Therefore, in this study, we hypothesize that fusion of cortical activation and functional connectivity network patterns can improve emotion recognition and classification.We propose a feature-level fusion approach to combine local cortical activations and functional connectivity network patterns.We estimate the cortical activations using power spectral density with Burg autoregressive model and evaluate the functional connectivity networks by utilizing phase locking value (PLV).
The remaining parts of this paper is organized as follows.Section II presents the experimental protocol and the method of analysis used in this study.Section III presents the results of emotion recognition based on cortical activations and functional connectivity network patterns.Section IV provides a detailed discussion on the findings and compares it with the state-of-the art of emotion studies, as well as highlights the limitations of the study and provides suggestions for future studies.Finally, section V conclude this paper.

II. METHODOLOGY A. PARTICIPANTS
Twenty-eight healthy young right-handed students from the American University of Sharjah (20 males and 8 female, age: 21±1.5 years, (mean±standard deviation)) have participated in this experiment.All participants reported normal or corrected-to-normal vision.None of them had a history of neurological or psychiatric illness and had no current or prior psychoactive medication use.They were asked to abstain from caffeine, exercise, energy drink and tobacco use for 24 hours before testing.Each participant was briefed about the study and gave her/his informed consent before the start of the experiment.The experimental protocol was conducted following the Declaration of Helsinki and approved by the IRB committee at the American University of Sharjah.

B. EMOTION-ELICITING STIMULUS
The emotion stimuli composed of 245 pictures taken from two publicly available image datasets, GAPED and OASIS [32], [33].Three groups of images (115-funny images, 70-neutral images, and 60-plain images) were used as stimuli.Funny pictures involved pictures of human and animal babies, neutral pictures involved pictures of nature and the plain-images mainly depicted inanimate objects; e.g., a picture of plain Book was used.The pictures were selected according to their valence and arousal scale.Funny images had high arousal and valence, neutral images had low arousal and high valence rating, and plain images had moderate arousal and valence.When the plain target image appears on the screen, the participant has to pose an acted smile and hit a response key.
There was a total of 245 trials in this experiment as shown in FIGURE 1.Each trial begins with a drift check of one second to ensure the focus is on the screen, followed by the image stimulus display.For every trial, an image was displayed on the screen for a maximum of 2 seconds and the participant had to give a keyboard response by hitting any of the three letter keys as shown in FIGURE 1.All image stimuli were presented on a 19 inch LCD display which was placed at a distance of 50 cm in front of the participant.All participants were asked to hit letter ''Q'' or ''P'' or ''N'' once and only whenever they felt their emotion had changed and have to act to produce certain emotional expressions in the form of fake smile (hitting Q letter), true smile (hitting P letter) and neutral expression by (hitting N letter), respectively.The experimental paradigm was implemented in the Psychology software, and the behavioral responses were recorded and used for evaluating the emotions.The behavioral responses of the participants indicated that, stimuli induced different emotions according to the arousal and valence scale.
This experiment lasted about 13 min on average.The number of trials also depended on the participant's rating speed.Note that, the order of images' presentation was semirandomized, with the condition that no two pictures from the same category appeared immediately one after the other.Three different markers were sent to mark the epochs of each type of images.

C. DATA ACQUISITION
Electrophysiological signals were recorded using 64 Ag/AgCl scalp electrodes arranged according to the standard 10-20 system (ANT waveguard system and ASA Lab 4.9.2 acquisition software, ANT Neuro, the Netherlands).In this study, the EEG data were acquired at a sampling rate of 500 Hz.The impedances of all EEG electrodes were maintained below 10 k , and referenced to the left and right mastoids, M1 and M2.FIGURE 2 shows the data acquisition and experimental set-up.
Summary of the Experimental Protocol: 1) Brief introduction about the task and experimental set-up.2) All participants were instructed to show emotional expressions to each of the three types of stimuli.For example, if the target image appears as funny, we expected a true/genuine smile, while neutral expression for neutral image and an acted fake smile for the plain image, plain book.3) If the participants did not find a picture funny, they would mark it as neutral.
4) All participants evaluate their emotions by filling emotion questionnaire.5) Behavioral responses showed that image stimuli induced different emotions.

D. DATA PROCESSING
EEG signals were preprocessed using custom scripts [34], [35] as well as the EEGLAB toolboxes (9.0.4) [36].The EEG signals were re-referenced to the common average reference [37] and segmented into target-related EEG epochs of 1500 ms.Baseline removal and DC offset were performed by subtracting the mean from the data.All EEG signals were band-pass filtered using a finite impulse response (FIR) filter with 0.1 Hz and 30 Hz cut-off frequencies.The 50Hz electrical power line interference was removed with independent components (ICs) using the CleanLine plug-in available on EEGLAB.The abnormal epochs were manually removed.Similarly, eye-blinks and eye-movements were removed manually by visual inspection as well as utilizing Independent Component Analysis (ICA) technique using the Infomax algorithm.Here, the components representing artifacts, such as eye blink, eye movements, and muscular activities were removed and the remaining components were used to reconstruct the clean EEG signals.Typically, only one or two independent components relevant to eye blinks or eye movements were removed for each subject.Finally, all EEG epochs were visually double checked to ensure the quality of the EEG data.Further analysis in this work would only include the correctly performed trials that had the artifacts mitigated in all channels (we use 180 trials in this study distributed equally across the three different emotions).

E. CORTICAL ACTIVATIONS
We estimate the cortical activation based on the power spectral density (PSD) with the autoregressive model (AR).It is simple and computationally efficient compare to other techniques [38].Moreover, AR modeling is preferred over other spectral estimation techniques, such as the Fast Fourier Transform (FFT), because of its superior resolution for short time segments.It is typically used in brain computer interface applications.AR modeling is well suited for EEG signals for several reasons.Firstly, EEG is non-stationary and must be evaluated using short time segments over which the data is assumed stationary.The length of the input process does not explicitly limit the spectral resolution of an AR model, and therefore, it is capable of providing superior resolution for short data segments.Secondly, EEG is essentially comprised of the superposition of mass single-unit activity through volume conduction.This may be considered as a filtered white noise process.This filtering process is the basis for AR modeling, and therefore, it is a reasonable approach to adopt for EEG modeling [39].Then, in order to ensure stationarity of the EEG signal, we tested the stationarity over different sliding windows of length 500, 750 and 2000 data points.We found consistent stationarity at window size of length > 500 data points.Therefore, we have chosen to take sliding windows of 750 data points across the 60 trials of EEG signals (i.e.we use 60 sliding windows of length 750 data points in each type of EEG signal).This number is large enough to show the stationarity of EEG signals and have been reported in previous EEG studies with a comparable data point [40], [41].
Note that, the model order (p) may influence the fitting results of the AR model.Several issues complicate the evaluation of AR model order.Different methods for estimating the AR coefficients can result in different optimal model orders.Additionally, band-pass filtering of signals tends to produce lower model order estimates when spectral content with considerable variance is removed from the signal.Furthermore, the length and sampling rate of the data can have a significant effect on model order estimates; higher sampling rates, and longer data segments may capture increased spectral content, thus requiring a higher model order [42].All of these issues greatly complicate the determination of the optimal AR model order and must be considered when generating a model.The most commonly considered order estimation methods include Akaike Information Criterion (AIC), Bayesian Information Criterion (BIC), and Final Prediction Error (FPE) [38], [43]- [46].In this study, AIC is used to determine the proper model order p (we used p = 10).Previous studies have also reported the effective AR-orders in EEG spectrum analysis using p = 2, 4, 6, 8, 10, 16, and 30 [38], [47], [48].
We evaluated the PSD based on autoregressive Burg method, by minimizing the forward and backward prediction error.We applied a sliding Hamming window with a length of 750 points (1.5s) for the spectral estimation as suggested by [49].The procedure consist of two steps: firstly, the AR model parameters are estimated from a clean EEG data sequence (x(n), 0 ≤ n ≤ N − 1, N = 750) for each EEG frequency band.In this paper, we take this strategy to perform PSD analysis of EEG waves in four clinical frequency bands, i.e., the delta waves (δ [0.1-4] Hz), theta waves (θ [4-8] Hz), alpha waves (α [8][9][10][11][12] Hz) and beta waves (β [12-30] Hz).Then, we use the autoregressive (AR) method for modeling the EEG data sequence x(n), as the output of a causal and discrete filter [50].The AR model used here is a linear regression of the current observation of the series against one or more prior observations of the series.AR model of order p for a zero-mean time series x(n) is written as: where x(n) is the EEG signal to be modeled, n refers to sample point, a(k) is the AR coefficients, ω(n) is the white noise which is independent of the previous points with variance σ 2 , and p denotes the number of previous time points used to model the current time point.We computed the AR coefficients a(k), k = 1, . . ., p for all EEG electrodes.For each trial, we then calculated the PSD from the AR parameters using Burg algorithm.Please see Appendix A for expressions used in this work for this purpose.The PSD values were then normalized between [0, 1], in which the value of '0' indicates low cortical activation and '1' indicates high cortical activation.The grand-average of the PSD is then presented in a topographical form to illustrate the cortical activations under the three different emotional expressions.

F. FUNCTIONAL CONNECTIVITY NETWORK
The functional connectivity network (FCN) is constructed by quantifying the coupling between pairs of electrodes using Phase Locking Value (PLV).Like the traditional coherence method, the PLV method calculates the correlation between two sets of EEG signals in different frequency bands.However, unlike the conventional coherence method, the PLV method deals with the phase characteristics of the signal.It also does not rely on the stationary-signal assumption.Therefore, PLV is considered more suitable for analyzing the functional coupling between EEG signals [51].
Several studies have highlighted the role of phase synchronization as a binding mechanism among segregated brain areas in attention, memory processes, and conscious perception [52]- [54].Studies of attention have reported a sharp decrease of phase synchrony during task execution and suggested that it may reflect the transitional state between two distinct cognitive processes [55].The same finding was also observed during the execution of an audiovisual perception task suggesting that it could reflect the disengagement of incoherent perceptual streams [56].Likewise, another study has illustrated that the task-related activity is mediated by distinct complex networks related to the phase desynchronization that configure their architecture dynamically during the task [57].Recent work has demonstrated that the phase synchronization of EEG signals has task-free biometric properties, which can be used for subject identification.This suggests that it can be a good candidate to quantify the coupling between electrodes in emotion studies.
Before computing the PLV, the clean EEG signals of all electrodes are band-pass filtered into four frequency bands similarly to the case of obtaining the PSD: δ (0.1-4Hz), θ (4-8Hz), α (8-13Hz) and β (13-30Hz) respectively to get their phase synchronizations.The PLV values were averaged over time samples using a 750 points segment.This seemed to be a good tradeoff between fast response time and robustness.
Consider a pair of EEG signals s 1 (t) and s 2 (t), that have been band-pass filtered to the frequency range of interest.The analytical EEG signals are obtained from s i (t) using Hilbert transform.Once the analytical signals are defined, the overall PLV is then computed by averaging over trials/segments [58] according to Equation (2).PLV can, therefore, be viewed as a measure of a trial-to-trial variability in the relative phases of two signals.This is very suitable for our type of experiment, an event-related paradigm.
here n indexes the trial number and N is the total number of trials (segments).Further details about the Hilbert transform and PLV relevant to our work are given in Appendix B.
The PLV takes values between [0, 1] with 0 reflecting the case where there is no phase synchrony and 1 where the relative phase between the two signals is identical in all trials.
The estimated PLV at each trial is then used for analysis and classification evaluation.In order to reduce the effects of volume conduction, we conducted statistical correction for each pair of electrodes {i, j} at every time point by wrapping the phase differences in the interval [−π , π ].We also tested if the mean distribution of the phase difference was significantly different from zero (we used t-test at 95% confident interval, p = 0.05 level of significance).We used only the PLV with phase-difference distribution significantly different from zero for further analysis.

1) ANALYSIS OF EEG CONNECTIVITY NETWORK
For each subject, the PLV is obtained in the four frequency bands for the three emotional states.The average data format of the connectivity matrix is 62×62×4×28, which represents the channel × channel × frequency band × subjects.
To analysis the connectivity network, we adopted graph theory analysis methods.The maximum number of connections in each network within K-nodes is measured using: N = K (K −1)/2.In total, we have 1431 pairs after excluding electrodes located within the center of the cortex, which have no lateralization.
In this case, each electrode is regarded as a node in a network, and the connection value is regarded as a weighted connection edge between two nodes.To reduce the effects of volume conduction, we subtracted the connectivity in-between different emotions.
We then extract the topological features using graph density and clustering coefficient from the normalized network.These metrics have been successfully used in graph theory studies [59], [60].

2) CLUSTERING COEFFICIENT (C)
Clustering coefficient C is a local density of connections used to quantify the number of relationships/connections between nearest neighbors of a node as a portion of the maximum number of possible links.

3) DENSITY (DEN)
Graph density indicates how many edges (pairwise connection between nodes) are inside the graph divided by the maximum possible number of edges between the vertices of the graph.In this work, we define the density of the graph as the ratio of the number of observed connections in the network to the number of possible links.The formulation and explanation of the clustering coefficients and graph density are given in Appendix C.

G. STATISTICAL ANALYSIS
To reveal the differences in brain responses to different emotions, we analyzed the differences between them in electrode-based using two-sample t-test.The t-test is measured between EEG power features as well as the phase synchronizations (PLV) for the three different emotions.The comparison was between features extracted from the True emotion versus Neutral emotion (T vs N), True emotion versus Fake emotion (T vs F) and Neutral emotion versus Fake emotion (N vs F) respectively.In each electrode, the differences were considered statistically significant if the p-value is less than 0.05 (p < 0.05).

H. FUSION ANALYSIS
The extracted features from the local cortical activations are fused with the features extracted from the dynamic functional network connectivity to form a union feature for the same recognition problem.The fusion here is based on concatenation of the two features (one from PSD and the other from PLV based on clustering coefficient and density).Such feature fusion helps in classification if different features are giving different but useful information about the problem.

I. CLASSIFICATION EVALUATION
The extracted features from the cortical activations, phase synchronizations before and after fusion in alpha and beta frequency bands within the three emotional states are used as an input to support vector machine (SVM) classifier (FIGURE 3).In this study, we used LIBSVM to build the SVM classifier with radial basis function (RBF) similar to our previous studies [61], [62].For each subject as well as for the average of all subjects, a ten-fold cross-validation (CV) procedure used to train and test the SVM classifier.In the ten-fold crossvalidation, each of the PSD features, PLV features, and the fused PSD+PLV features split into ten subsets (10 crossvalidation).Nine subsets were used to train the SVM classifier, and the remaining one subset was used for estimation of classification accuracy, sensitivity, specificity and the area under the receiver operating characteristic curve (AROC).The classification of emotions was conducted in the form of one-vs-one such as genuine/true-vs-neutral; genuine-vs-fake, and neutral-vs-fake, for each individual subject as well as for the average of all subjects.Note that, we average the features of all subjects in the early stage of feature extraction at each trial.

III. RESULTS
In this section, we present the results of emotion classification based on cortical activations, connectivity network patterns  and after fusion of cortical activations and connectivity network patterns.Although, we presented the results of emotion at all frequency bands, only the highly associated frequency bands with emotions used for classification evaluations.

A. CORTICAL ACTIVATIONS ANALYSIS
The cortical activations based on the average weight normalized PSDs in the three emotions for each frequency band are calculated and reported in the form of topographical maps covering the entire brain with EEG electrodes.These are shown in FIGURE 4.
BETA: The results of PSD in the beta frequency band at the true/genuine emotion showed high cortical activations on the frontal, temporal, and parieto-occipital brain areas.During neutral emotion, the left frontal, parietal and occipital regions showed moderate cortical activations.During fake emotion, high activations shown on the left midline, and left parietooccipital brain areas.
ALPHA: The true/genuine emotion showed high cortical activations on the frontal, and parieto-occipital regions similar to that at beta frequency band.However, during the neutral emotion, only the left midline and lateral- ized parieto-occipital brain areas showed moderate cortical activations.During fake emotion, the left frontal, and parieto-occipital regions showed higher cortical activations compared to the other regions.
THETA: The true emotion showed high cortical activations on the left frontal, and parieto-occipital brain areas.During neutral emotion, only the left frontal, and parieto-occipital regions showed moderate cortical activations.The fake emotion showed high cortical activations on left midline and left parieto-occipital brain areas compared to the other regions.
DELTA: The true/genuine emotion showed high cortical activations on the frontal, and parieto-occipital brain areas.During neutral emotion, only the left frontal and parietooccipital regions showed moderate activations.During fake emotion, the left frontal and left parieto-occipital regions showed higher cortical activations compared to the other regions.
The statistical analysis of cortical activations between the three emotions in beta, alpha, theta, and delta frequency bands is shown in FIGURE 5 in the form of t-maps.The value of t >= 2.1 indicate that the difference between emotions is significant, p < 0.05.Lateralized electrodes that have high t-value were used for classification evaluation.Table 1 shows the selected pair of electrodes for emotion classification evaluation at the cortical activations and network connectivity patterns.

B. FUNCTIONAL CONNECTIVITY ANALYSIS
In this section, we present the results of brain functional connectivity network (FCN) patterns under three different emotional expressions in the four frequency bands.The normalized FCN on inter and intra-hemispheres between different emotions are shown in FIGURE 6.Only the subtracted grand-average FCNs across all subjects in the different emotions within the four frequency bands are reported.
BETA: The FCNs in the true/genuine emotion was associated with more synchronization than neutral and fake emotions, mainly at the right frontal-to-parietal, frontal-totemporal, and left parieto-occipital regions.The FCNs at the fake emotion showed high synchronization in the left hemisphere mainly at the frontal-to-parietal, frontal-to-occipital and right midline compared to that of neutral emotion.
ALPHA: The FCNs in the true/genuine emotion showed high synchronization in the left frontal-to-parietal and frontal-to-occipital brain areas compared to that of neutral and fake emotions.The FCNs in the fake emotion showed high synchronization in the left frontal-to-occipital and right frontal-to-central compared to that of neutral emotion.
THETA: The FCNs in the true/genuine emotion showed moderate synchronization in the left frontal-to-occipital, frontal-to-temporal, and parieto-occipital and right frontalto-occipital compared to neutral and fake emotions.Note that, the high synchronization was on the right hemisphere specifically the right frontal-to-parietal and right frontal-tooccipital compared to neutral and in the left hemisphere as specifically, the frontal-to-occipital, frontal-to-parietal and frontal-to-temporal compared to the phony emotion.The FCNs in the fake expressions showed high synchronization in the frontal-to-parietal and frontal-to-temporal regions compared to neutral emotion.
DELTA: The FCNs in the genuine emotion showed high synchronization at frontal-to-parietal, and frontal-to-occipital regions compared to that of neutral and fake emotion.However, the FCNs in the fake emotion showed high synchronization only over the frontal-to-occipital and left frontal-toparietal regions compared to neutral emotion.
The statistical analysis of FCN showed that beta and alpha frequency bands were highly sensitive to different emotions in this study.Therefore, the graph density and clustering coefficients were extracted from these two bands only.FIG-URE 7 shows the network density and clusters for the three emotions.The results show that true emotion has a higher density, as well as high clustering coefficients, compared to that of neutral and fake emotion.Additionally, the results show that as density increases, the clustering coefficient also increases.This is because a graph with higher density has more triangles compared to a spare graph.The variations in density and clustering coefficient seem consistent across the alpha and beta frequency bands.

1) EEG CHANNEL SELECTION FOR INDIVIDUAL AND GROUP ANALYSIS
In this study, we utilized various EEG channels/electrodes that cover different regions of the brain while taking into consideration the effects of emotion on brain asymmetry.The selection of EEG channels was based on their significant response in differentiating between the three emotions, p < 0.05.Several studies have reported frontal, prefrontal, temporal, parietal and occipital regions as the most sensitive  brain regions to emotional responses [63]- [65].Likewise, we utilized nine-pairs (nine asymmetries) of electrodes that cover the prefrontal, frontal, parietal and occipital regions of the brain for classification evaluation, as listed in Table 1.These pairs of electrodes, in fact, were highly sensitive to different emotions in the cortical activations as well as in the functional connectivity network patterns as demonstrated in FIGURE 4 to FIGURE 6.The selection criteria were applied to beta and alpha frequency bands since they were highly associated with emotions.

C. CLASSIFICATION ANALYSIS
The results of the classification performance in differentiating between three different emotions are reported in terms of accuracy, sensitivity, specificity, and area under the receiver operator characteristic curve (AROC).The result of classification is presented based on combination of alpha and beta features which has been reported in literature [66] to give better performance than individual frequency band features  under the same conditions.The distribution of classification performance across subjects is presented in boxplots with violin graphs for easy visualization.FIGURE 8 to FIGURE 10 show the distributions of classification performance of different emotions using individual PSD, PLV and fusion  of both; PSD+PLV.Note that the fusion of PSD+PLV significantly enhance the classification performance compared to individual feature in all the evaluation metrics for all subjects.The results of classification performance using PLV outperformed using PSD in all the evaluation metrics.Table 2 shows the results of classification performance when using individual subject analysis represented by the mean with standard deviation.Table 3 shows the results of classification performance when using the average features of all subjects.The performance of average-subjects analysis outperforms individual analysis, indicating that individuals perceived the emotions differently.Additionally, the classification results show that the specificity score higher than sensitivity/accuracy in the individual PSD, PLV feature vector as well after fusion of PSD+PLV feature.Overall, the results of the classification accuracy, sensitivity, specificity, and AROC significantly improve after feature-fusion compared to individual features, suggesting that connectivity networks complement the cortical activations.Note that, the yellow colors in Figure 8 to Figure 10 show the classification scores when the PSD features are used, green colors show the classification scores using PLV features and the grey colors show the classification scores after fusion of PSD+PLV features, respectively.The red horizontal line in each of the boxplot and violin shows the median of the score and the smooth black line indicates the mean of the score in the classification performance.

IV. SUMMARY AND DISCUSSION
This study aimed to improve emotion classification based on the fusion of cortical activations with functional connectivity network patterns.The study utilized power spectral density based on the autoregressive model to estimate the cortical activations and the phase locked value to evaluate the functional network connectivity.It was performed on twenty-eight subjects that underwent three different stimuli.The stimuli were intended to induce true/genuine, neutral and fake emotions.The study then performed statistical analysis on the features extracted from the PSD/PLV of each subject individually as well as on the average of all subjects.It showed significantly different patterns between emotions.The high-frequency components in beta and alpha bands were found to be highly sensitive to different emotions.The classification results later showed that fusion of cortical activations with functional connectivity network patterns outperformed individual features in all the evaluation metrics, p < 0.05.Up to date, this is the first study to combine cortical activations with connectivity network patterns for classifying three different emotions and to evaluate their performance in term of accuracy, sensitivity, specificity, and the area under characteristics curve using the EEG signals.
In this study, the procedures of inducing three different emotions were validated with self-evaluation and behavioral responses.All subjects showed high arousal and valence when exposed to funny images, low arousal and high valence to neutral images, and moderate arousal and moderate valence when plain-images appeared with high accuracy (in line with [32], [33]).The results of high-density EEG signals indicate that the content of emotional stimuli induces different cortical activations as well as different connectivity network patterns.The local cortical activations in our results appeared mostly in the frontal and parieto-occipital regions.The results of cortical activations in the frontal and parietal regions in this study indicate emotions modulated attention in which participants were mentally engaged to the task/stimuli.Although we only analyzed the cortical activations at beta and alpha frequency bands, other frequency bands showed activations as well but not as high.The higher cortical activations in the high-frequency components are due to their associations with emotions and cognitive process.Previous emotion studies have shown similar activation patterns at higher frequency components of beta and gamma bands [17], [19], [28], [63], [67].In line with that, it may be conjectured that the activations at higher frequency components reflect emotion regulation [68]- [70].
Similarly, the functional connectivity network patterns at beta and alpha frequency bands showed dense network when participants viewed images that induced true/genuine emotion compared to that at neutral and fake emotions.The significant differences in the network patterns were mainly on the frontal-to-parietal and frontal-to-occipital brain areas.However, in fake emotion, less connectivity network patterns were found compared to that of neutral emotion.Nevertheless, different asymmetries were found across different frequencies as showed in FIGURE 6.For example, at a beta frequency band, the genuine emotion showed high connectivity in the right hemisphere (specifically in the frontal-occipital, frontal-totemporal) compared to neutral and fake emotions.Similar observations have been reported for frontal-to-temporal site while participants watching film clips with positive emotions [31].Likewise, at the alpha frequency band, the connectivity patterns in the left hemisphere were highly connected particularly in the frontal-to-parietal and frontal-to-occipital brain regions.It also showed that fake emotions produced dense networks within the left hemisphere compared to the right hemisphere in beta and alpha frequency bands.The left dominance in fake emotion is in line with previous studies that used music to induce different emotions [71], [72].The increase and decrease in the connectivity network patterns in our study are in line with previous emotion studies that showed happiness produced stronger activation in the dorsolateral prefrontal cortex, cingulate gyrus, and inferior temporal gyrus, see review [73].
Taken all together, the results of our study showed that true/genuine emotion produced great connectivity patterns compared to fake emotion in all the frequency bands.It was also found that at high cortical activations the connectivity showed dense or wide network.Similar result was found in a previous emotion and working memory study [74].Although we found different cortical activations and connectivity patterns in the three emotions, it is very important to involve negative emotions to establish functional connectivity as an index for emotion processing and regulation [75].
The results of classification performance demonstrated that fusion of cortical activations with functional connectivity network patterns outperforms individual feature sets with p < 0.05.Considering, the average of all subjects, we achieved 90.3%, 88.52% and 78.82% classification accuracy between true/genuine versus neutral, true/genuine versus fake, and neutral versus fake emotions respectively, as demonstrated in Table 3.Similarly, in the subject-level analysis, we achieved 87.71%±5.28%,86.35%±5.22%,and 77.72%±4.45%,accuracy between true/genuine versus neutral, true/genuine versus fake, and neutral versus fake emotions respectively, as demonstrated in Table 2. Our fusion results showed an average improvement of 7.10%, compare to PSD alone, and 4.01% compared to PLV alone in term of classification accuracy.The improvements in the classification performance was also found across all the proposed evaluation metrics (accuracy, sensitivity, specificity, and the AROC on average of 6.84%, compared to PSD, and 4.1% compare to PLV) indicating that, connectivity network patterns provide complementary information to the cortical activations.The improvements in the classification performance in our study were in line with previous fused emotion studies that combined cortical activations of EEG with facial features [17], [18], and with eye tracking data [22], [26], [76], [77].Overall, the average classification performance in our study outperforms the other published results of emotion classification studies (achieved classification accuracy ranges between 58% and 85%) [20]- [30], [78]- [81] based on ENTERFACE'06, MAHNOB-HCI, SEED, and DEAP datasets.
Note that, it is important to consider the experimental protocol and the assessment method differences between studies.Our approach may, however, have high sensitivity to detect emotions due to the use of high-density EEG neuroimaging.
Limitation of the Study: This study improved emotion classification and recognition well above the chance level based on the fusion of cortical activation and functional connectivity network patterns.However, it poses some limitations, and further studies will be needed in some aspects.Firstly, the emotion induction stimuli in this study were based on viewing images.Nonetheless, some experimental studies utilized film clips, facial expressions, and music for emotion induction.Literature has reported that different type of stimuli led to different brain activation [82].Using a combination of various stimuli would be necessary to understand the relationship between the neural mechanism and emotion.Secondly, this study evaluates the functional connectivity network using phase locking value.Other multivariate methods such as Partial Directed Coherence (PDC) and Directed Transfer Function (DTF) with Granger causality [83] may be applied in future emotion studies to estimate the directed brain functional connectivity.These methods gave more quantitative data as they reflect the directions of the information flow in the network.Thirdly, it is essential to consider the effects of volume conduction when evaluating the EEG connectivity network at the scalp level [84].One such method suggested by [85] could be an excellent alternative to solve this problem.Another method is to combine high-density EEG with another neuroimaging technique that has good spatial resolution such as fMRI or fNIRS.Furthermore, this study was conducted on young healthy subjects; old people may perceive emotion stimuli differently.Future studies on AR model may use a regressor to include the current data in the prediction model.

V. CONCLUSION
This study improved the classification accuracy, sensitivity, specificity and area under the receiver operator characteristics curve on classifying three different emotions based on feature fusion of cortical activations with functional connectivity network patterns.The study achieved up to 90.3%, 88.52% and 78.82% classification accuracy between the three different emotion classification tasks.The results also showed different activation patterns in the three emotions at all the frequency bands.Nonetheless, results in high-frequency components at alpha and beta bands were highly associated with emotions.
The current results demonstrate the potential of the fused cortical activation at the single electrode level with the dense functional connectivity network patterns and suggest it for future emotion studies.where êp is the total least squares error and p is the order of AR model, p = 10 in our study.

APPENDIX B
Formulation and explanation of analytical EEG signals using Hilbert transform and Phase locked value (PLV).
Consider a pair of EEG signals s 1 (t) and s 2 (t), that have been band-pass filtered to the frequency range of interest.The analytical EEG signals z i (t): = A i (t)(cos(φ i (t)) + j sin(φ i (t))) (B.1) for i = {1, 2} are obtained from s i (t) using Hilbert transform according to the following equation: where HT (s i (t)) is the Hilbert transform of s i (t) define as : the φ is the difference in phase between the two time series, and P.V means that the integral is taken in the sense of Cauchy principal value.Once the analytical signals are defined, the relative phase then computed as: Formulation and explanation of global network metrics.The clustering coefficient (C) is an index of local structure of a graph.For a network graph, the C of node i is defined as: where E denotes the number of unique edges in an undirected graph with no self-loops and N is the network size.

FIGURE 2 .
FIGURE 2. EEG data acquisition and experimental set-up.

FIGURE 3 .
FIGURE 3. Schematic of feature level fusion and classification of emotion.

FIGURE 4 .
FIGURE 4. Topographical maps of cortical activations of four frequency bands under three different emotions.

FIGURE 6 .
FIGURE 6. Dynamic functional networks (averaged over all participants) in different frequency bands calculated by subtracting the networks corresponding to True, Neutral and fake.

FIGURE 7 .
FIGURE 7. Network Density and Clustering Coefficients of true, neutral, and fake emotions within: A) in Beta Band and B) in Alpha Band, ( * p < 0.05.* * p < 0.01).

FIGURE 8 .
FIGURE 8. Classification Performance of true versus neutral emotion a) demonstrates the performance accuracy; b) represents the sensitivity; c) shows the specificity of the classifier and d) demonstrates the area under the receiver operator characteristics curve.Note that the yellow violin box demonstrates the performance evaluation using PSD while the green violin is for the PLV and the gray violin demonstrated the classification performance after fusion of PSD with PLV.

FIGURE 9 .
FIGURE 9. Classification Performance of true versus fake emotion a) demonstrates the performance accuracy; b) represents the sensitivity; c) shows the specificity of the classifier and d) demonstrates the area under the receiver operator characteristics curve.Note that the yellow violin box demonstrates the performance evaluation using PSD while the green violin is for the PLV and the gray violin demonstrated the classification performance after fusion of PSD with PLV.

FIGURE 10 .
FIGURE 10.Classification Performance of neutral versus fake emotion a) demonstrates the performance accuracy; b) represents the sensitivity; c) shows the specificity of the classifier and d) demonstrates the area under the receiver operator characteristics curve.Note that the yellow violin box demonstrates the performance evaluation using PSD while the green violin is for the PLV and the gray violin demonstrated the classification performance after fusion of PSD with PLV.

Formulation
and explanation of Power spectral density obtained from the autoregressive model Mathematically, we calculated the PSD from the AR parameters using Burg algorithm for each trial according to:

c
ik c il c kl k =i l = i l = k c ik c il (C.1)where c ik , c il is the linkage strength between node i and node j estimated by PLV, and k, l indicate any two different nodes.Graph density is defined as the ratio of the number of observed connections in the network to the number of possible links.DEN = 2E N (N − 1) (C.2)

TABLE 1 .
Selected EEG electrodes for classification analysis.

TABLE 2 .
Performance evaluation of Individual Subjects, mean±standard deviation.

TABLE 3 .
Performance evaluation for the Average of all subjects.