Filter Bank Convolutional Neural Network for SSVEP Classification

Harmonics in electroencephalogram (EEG) caused by visual stimulation are the main basis of classification of steady-state visual evoked potential (SSVEP). However, the correlation of various harmonics, which could improve the classification performance especially when evoked EEG components are much weaker than spontaneous EEG components, has not been take into consideration in the design of classifier in previous studies. In this study, we proposed a filter bank convolutional neural network (FBCNN) method to optimize SSVEP classification. Three filters with passbands covering each harmonic of SSVEP signals are used to extract and separate the corresponding components, and the information from them are transformed into frequency domain. Subsequently, we introduce a novel convolutional neural network (CNN) architecture with three parallel CNN channels to extract and learn the harmonic features in passbands, and conclusions on the correlation among harmonics can finally be made by pair-add-up operations and dimension reductions to weigh the feature vectors. The proposed FBCNN is evaluated on two public datasets (Dataset1: 12-class, 10 subjects; Dataset2: 40-class, 35 subjects) to compare with other methods. The experimental results illustrate that FBCNN method improves the performance of CNN-based SSVEP classification methods and has a great potential to be applied in SSVEP-based BCI.


I. INTRODUCTION
Brain-computer interface (BCI) based on electroencephalogram (EEG) measures EEG signals in a noninvasive way, extracts the specific features, subsequently, converts them into the commands of the equipment [1]. BCI provides a novel hand-free communication channel to control peripheral devices by realizing the interaction between human brain and machine, thus could facilitate lives of the disabled, help training of stroke patients with rehabilitation and even control computer games [2]. Among different EEGbased BCIs, e.g. motor imagery (MI) and event-related potentials etc., steady-state visual evoked potential (SSVEP) possesses the advantage of high information transfer rate (ITR) and few training times [3] and thus is widely used in the context of human-machine interaction.
In SSVEP, the visual stimuli with different flicker frequencies is applied and the consequent oscillations could occur in the visual cortex, which could be further detected in EEG signals in the form of the strong amplitude of corresponding frequency and harmonic. Based on the observed pattern of the detected EEG signals, the recognition algorithm could be used to find out target stimulus [3]. Generally, the performance of SSVEP-based BCI is mainly determined by three factors, namely, stimulus presentation, multiple target coding and target identification algorithm [4].
Practically, the target identification algorithm of SSVEP-based BCI can be divided into three categories: training-free methods, user-specific or user-dependent training methods and user-independent training methods [5]. Training-free methods compute the relevance between the detected signals and the potential stimuli, and hence directly determine the classification results. These methods mainly include power spectral density analysis (PSDA) [6], canonical correlation analysis (CCA) [7] and minimum energy combination (MEC) [8]. Among these methods, CCA, the most prevalent training-free method, is always treated as the baseline algorithm for SSVEP detection [9]. It aims at finding the potential correlation between the detected EEG data and a set of sinusoidal reference templates corresponding to the stimulus frequency.
In contrast with the training free method, the training methods, either user-dependent (UD) or user-independent (UI) method, incorporate the features extracted from the trail data to improve the classification accuracy [5]. The difference between UD and UI training methods lies in the training and testing strategy, which can be observed in Fig.  1. As for UD training methods, the trained model is generated from the trail data from one person, and is only suited for this individual in the classification step. However for UI training methods, a generalized model generated by the training data from multiple participants is applied to extra users, i.e., arbitrary new users have access to BCI equipment with no collection of their own training data. For example, Combination method [10], Individual Template CCA (IT-CCA) [11] and Multi-way CCA (MwayCCA) [12] are considered as the UD methods and Filter Bank CCA (FBCCA) [13] is the typical UI methods. The design strategies of filter bank will be described in the following paragraphes.
Recently, deep learning has developed rapidly in view of its prominent capabilities of feature extraction and learning [14], providing new insight to the classification and the detection of EEG-based BCI. In fact, the deep learning based methods tend to outperform traditional methods in most fields including EEG signal detection [15], face recognition [16] and cross-media retrieval [17]. The advantages of convolutional neural network (CNN) over standard deep neural network (DNN) can be seen in prior researches [18], and the time delays term in neural networks has also been explored in depth [19]. In short, CNN structure plays the most popular role in different deep learning-based BCI algorithms [20].
The CNN-based methods in SSVEP can be categorized in many perspectives. For example, the input data of the algorithms, which could impact the performance of the classifiers, can generally be classified into two broad categories: time domain data and frequency domain data. When time domain signals used as input, wider and deeper convolution operations are generally required in feature extraction step, while the neural networks with the input of frequency domain data don't need too many convolution operations [9], [15], [24]. Given that significant frequency and phase characteristics in SSVEP signals usually remain stable during stimulation, Fast Fourier Transform (FFT) [9], [21], [22], [23] is widely applied to transform SSVEP signals into the frequency domain before feeding into classifiers. According to the reported researches, the methods using frequency domain data as input tend to present better performance [9]. The other main difference lays in the diverse training strategies for the CNN-based methods. UD training procedure uses training and testing data from the same participant, whereas data from different participants are used for training and validating respectively in UI training procedure. From the related studies, classifiers based on UD training procedure perform better than that based on UI training procedure due to the generalization ability of classifiers [9].
Although the CNN-based SSVEP classification algorithms has been widely recognized for its end-to-end characteristic, the prior knowledge for SSVEP, for example, the relevance among different SSVEP harmonics, has not received deserved attention. In some cases, as is shown in Fig. 2, it is necessary to analyze the correlation among different harmonic components of the evoked EEG signals to optimize classification. Since the spontaneous EEG components of the subjects are much stronger than the SSVEP evoked EEG components in light of inattention, individual difference or interference caused by environment etc, classifiers pay more attention on spontaneous EEG signals while neglecting the harmonics with weaker signal amplitude. Therefore, we put forward a novel input structure of the classifier and a new CNN architecture.
Our design strategy of the proposed filter bank differs from previous studies. As is shown in Fig. 3, we compare three different filter bank schema. Fig. 3(a) illustrates one typical filter bank structure using plenty of narrow band-pass filters with equally spaced bandwidths to extract independent frequency components from SSVEP signals. Fig. 3(b) shows another typical structure owing multiple filters with different bandwidths to cover all harmonics [13]. These methods aim to concentrate the feature analysis on each independent frequency band, ignoring the correlation among harmonic components. In our method, as is shown in Fig. 3(c) and Fig.  4, three filters with passbands covering the 1st, 2nd and residual harmonics of the input SSVEP signals respectively are used to extract each harmonic component before FFT operation. The 1st harmonic represents the stimulation frequency range. The detailed example of filer bandwidth can be found in the experiment section. Moreover, our neural network utilizes three parallel CNN channels to extract and learn the harmonic component information in each passband, other than mixed harmonic components in the CNN structure proposed in [9]. By means of pair-add-up operations and dimension reductions on the output features of each CNN channel, the feature vectors of each harmonic are weighted to find the correlation among them. Finally, the classification results are output through fully connected layer. By foregoing operations, we learned not only the feature of harmonics but also relevance among them, which receive less attention when single band used in CNN.
In order to compare with the method demonstrated in [9], we also implement our method in both UD and UI training procedure. Additionally, to verify the feasibility of the model and the fairness of the results, we test the proposed model using two public datasets: 1) Dataset1: a publicly available twelve class SSVEP dataset with 10 participants [10]; 2) Dataset2: a publicly available forty class SSVEP dataset with 35 participants [25]. Since Dataset2 was not used in [9], we reproduce the architecture proposed in the paper for comparison.
The paper is organized as follows. Section II outlines the proposed SSVEP classification method. Section III details the information of two datasets and training parameters. The performance of the proposed method is presented in Section IV. Section V discusses the results of the comparison. Finally, Section VI concludes the paper and prospects for the future work.

II. METHODS
In view of strong correlation between harmonics [13], different SSVEP harmonic components are used in the classification of SSVEP. To extract various harmonics for feature learning, filter bank is introduced to separate each harmonic component. Considering the computational complexity and classification accuracy [13], [26], three bandpass filters are selected to preprocess the EEG signals before FFT operations, followed by a corresponding neural network to classify the SSVEP targets, and Fig. 4 displays the input structure of the proposed method. The whole algorithm structure is called filter bank convolutional neural network (FBCNN). In the proposed method, three independent CNN subnets are used to extract features and weight contributions of different harmonics, and the attention mechanism offers an enhancement of the weighted features. The follow-up pairadd-up operations and the fully connected layers are used to fuse different harmonic features and output the classification results.

A. DATA PROCESSING
To compare the performance of various classification algorithms, different data preprocessing methods are employed in two datasets. The data preprocessing method for Dataset1 and Dataset2 is based on [9], [10], [24] and [25] respectively for comparison, and we reproduce the CNN [9] structure for Dataset2. In detail, a 4th order Butterworth band-pass filter is used to remove the artifacts which may exist in EEG signals for CCA [7] and CNN [9]. In our case, the EEG data are filtered by three 4th order Butterworth band-pass filters with different frequency bands to separate the harmonics. Then, the filtered EEG data of each SSVEP trail is divided into non-overlapping segments with 1s time window (TW).
As in the previous studies, the results of FFT for each 1s segment contain two parts: magnitude spectrum and complex spectrum [9]. The output of FFT is shown as follows: where x indicates the input time domain segment. The magnitude spectrum mag X can be calculated by: And for complex spectrum comp X , the real part and imaginary part of the FFT output components are concatenated into a single vector as: The magnitude spectrum only takes the magnitude information of the FFT results, without the phase information [9]. However, previous studies have shown the importance of phase related information presented in the SSVEP-based BCIs [24], [28], [29], [30]. Hence, it is necessary to use the complex spectrum which contains related information of both magnitude and phase simultaneously. The magnitude input mag I can be defined as:

B. FILTER BANK CONVOLUTIONAL NEURAL NETWORK
We propose a SSVEP classification method which combines filter bank and CNN structure. As shown in Fig.  3, three filters with different bandwidth are employed in order to fulfill the fine-grained feature learning of the SSVEP data. The bandwidth of Filter1 covers the first harmonic frequency range of the SSVEP, and Filter2 is for the second harmonic, Filter3 is for the rest. Three CNNbased subnets are introduced to compose the network of FBCNN for extracting and weighting features of different SSVEP harmonic components. Fig. 5 shows the network architecture of FBCNN. Each subnet contains 6 layers: an input layer, three convolutional layers, a Squeeze-and-Excitation (SE) block and a flatten layer. The output structure consists of three add layers and fully connected layers followed by subnets. The complete network has an additional concatenate layer and an output full connected layer.
The dimension of the input layer is ( +2)x the number of frequency components extracted from FFT result. In FBCNN, the preprocessed data of the first two EEG channels are repeated to achieve entirely convolution in the Conv1 which uses the "valid" padding mode. The input layer composition is shown as follows: is followed with the aim of enhancing the representational power of the network. SE block explicitly models the inner dependencies among channels of convolutional features to acquire preferable representation [31]. Fig. 6 shows the schema of the SE block, the input feature maps have the dimension of xx H W C which represent the height, width and channels of the feature maps respectively. The parameter r is the reduction ratio of the dimensionalityreduction layer, and the ratio is 8 in this study. The input feature maps are firstly passed through a squeeze operation to produce a channel descriptor by aggregating, then followed by an excitation operation to produces a collection of per-channel modulation weights [31]. The scale layer of the SE block rescales the feature maps by: where c u is the feature maps, and c s is the scalar. The flatten layer of each subnet is used to compress the output feature vectors of SE block to one-dimension. Subnets follow by layers to perform pair-add-up operations and fully connected layers to reduce dimension for weighting the feature vectors of each harmonic. And the number of units 6* class N output from fully connected layers equipped with the rectified linear unit (ReLu) activation function is in line with that of SSVEP classes. Then, the outputs of three CNNs are connected by a concatenate layer, and finally a fully connected layer possessed with the softmax function is used to output class N units corresponding to the probability of each SSVEP class. Batch normalization, ReLu activation function and Dropout are performed on Conv1, Conv2 and Conv3 layers of each CNN channel of the FBCNN network. For neural networks, Batch normalization and Dropout are commonly applied to enhance the generalization performance and calculation speed [23], [32], [33].

C. CONVOLUTIONAL NEURAL NETWORK
The architecture of CNN is shown in Fig. 7, which was proposed in the earlier study of SSVEP classification [9]. The CNN is composed of four main layers, an input layer with the dimension of  resolution .Thus, for Dataset1 the scale is 1x10 [9], and for Dataset2 the scale is1x15 . The scale, a fixed value of 1x10 in the earlier study [9], did not function effectively compared to 1x15 on account of different sampling frequency and FFT resolution when we reproduce the CNN structure on Dataset2. Furthermore, the kernel size of Conv2 has been changed into1x15 in order to ensure fairness.
Batch normalization and ReLu activation function are performed on the outputs of Conv1 and Conv2, and Dropout is used to avoid overfitting.

D. CANONICAL CORRELATION ANALYSIS
The training-free method CCA is introduced as a baseline to perform on the same EEG datasets. CCA, a multivariate statistical analysis method, reflects the underlying correlations between two groups of indicators. In previous SSVEP-based BCIs studies, CCA was used to calculate the correlations between EEG signals and reference signals corresponding to SSVEP classes [9], [10], [24], [25]. The canonical correlation coefficient ( , ) xy  of CCA is defined as: where X and Y are input EEG signals matrices and reference signals matrices respectively, and x , y stand for the linear representation of X , Y respectively.
x w and y w indicate the linear coefficient vectors. In this paper, sinusoidal signals are used as the reference signals Y to perform the classification in an unsupervised way. The reference signals Y is defined as: sin (2 ) cos (2 ) sin ( where f corresponds to the target frequency of the SSVEP, and h N is the number of harmonics.

E. ALGORITHM EVALUATION
One-way repeated measures analysis of variance (ANOVA) is applied to evaluate the results of CAA, CNN and FBCNN classification methods on both datasets, and the classification accuracy of each method is input as the response variable. The statistical significance level is 0.05 for analysis and comparison. The CNN and FBCNN are compared on every subject in the two datasets using both UI and UD training approaches. Moreover, the information transfer rate (ITR) of each method is calculated as follows: where N and T are the number of SSVEP classes and the average time for a selection , and P is the classification accuracy.

A. DATASET1 DESCRIPTION
The twelve stimuli are arranged on a 27-inch LCD monitor flashing at frequencies step of 0.5Hz ranging from 9.25Hz to 14.75Hz, with the corresponding phases ranging from 0π to 1.5π in steps of 0.5π [10]. *Values used directly from [9] EEG data are recorded from ten healthy subjects (9 males and 1 female, mean age: 28 years) with normal or corrected-to-normal vision using a BioSemi ActiveTwo EEG system (Biosemi, Inc.) with the sampling rate of 2048Hz. All subjects are seated in a comfortable chair in a dim room, 60 cm in front of the LCD monitor. The experiment consists of 15 blocks for each subject, and a block contains 12 trails corresponding to all 12 targets in a random order. A red square appears for 1s at the position of the target stimulus at the beginning of each trial. Participants are asked to shift their gaze to the target within the 1s duration. All stimuli start to flicker simultaneously for 4s after the red square disappears. Participants are instructed to avoid eye blinks during the 4s stimulation process.

B. DATASET2 DESCRIPTION
An open 40-target dataset for SSVEP-based BCIs conducts a cue-guided target selecting task. 40 characters are presented on a 23.6-in LCD monitor and the viewing distance to the LCD monitor is 0.7 meters [25]. The frequencies of 40 characters rang from 8Hz to 15.8Hz with the step of 0.2Hz, and the phase step is 0.5π. 40 targets are coded using a joint frequency and phase modulation (JFPM) approach [27]. EEG data are acquired using the Synamps2 EEG system (Neuroscan, Inc.) from Thirty-five healthy subjects (17 females, aged 17-34 years, mean age: 22 years) with normal or corrected-to-normal vision. The sampling rate is 1000 Hz. Each trial lasts 6s in total. At the beginning of each trail, subjects are asked to shift their gaze to a red square at the target location during 0.5s cue duration as soon as possible. Then, all stimuli start to flicker on the LCD monitor simultaneously for 5s after the cue. The blank in the LCD monitor lasts for 0.5s following the stimulation. The experiment includes 6 blocks for each participant, with one block containing 40 trials corresponding to all 40 targets indicated in a random order. A total of 240 trials are conducted on each participant.

. (a) Comparisons of the average classification accuracies and (b) the average ITR on Dataset1 for different methods with 1s data length. Error bar in each method indicates the standard deviation among all subjects.
(a) (b)

FIGURE 9. (a) Comparisons of the average classification accuracies and (b) the average ITR on Dataset2 for different methods with 1s data length. Error bar in each method indicates the standard deviation among all subjects.
VOLUME XX, 2017 9

C. DATA PREPROCESSING PARAMETERS
In Dataset1, the data of all the 8 channels are filtered by a 4th order Butterworth band-pass filter between 6Hz and 80H for CCA [7], while by three 4th order Butterworth band-pass filter in our proposed method FBCNN. The bandwidth of three filters are 6Hz-16Hz, 16Hz-32Hz and 32-64Hz respectively. Then, each 4s trail is divided into 1s non-overlapping segments. Considering the visual latency, a time delay of 135ms is added in the extraction [10], and the frequency components converted by FFT with a resolution of 0.25Hz are extracted between 5Hz and 55Hz for FBCNN methods. In Dataset2, O1, Oz, O2, PO3, POz, PO4, Pz, PO5 and PO6 are used as the input channels for all methods. For CCA [7] and CNN [9], the bandwidth is between 6Hz and 64Hz; for FBCNN, three band-pass filter are used to filter the data, and the bandwidth of three filters are 6Hz-18Hz, 14Hz-36Hz and 28 Hz-64Hz respectively. The first 140 milliseconds of data are removed because of the visual latency [25]. Finally, the filtered 5s EEG data of each trail is divided into 1s non-overlapping segments. The frequency domain signals transformed by FFT with a resolution of 0.2Hz are extracted between 5Hz and 55Hz for both CNN [9] and FBCNN.

D. TRAINING PROCESS
A normal distribution with a mean of 0 and a standard deviation of 0.01 is used to initialize the convolutional layers and fully connected layers of the FBCNN and CNN network. Both networks are trained by the stochastic gradient descent (SGD) optimization algorithm with the momentum of 0.9, and the categorical cross-entropy loss function with backpropagation technique is introduced to achieve the difference between predicted value and real value. The hyper parameters are obtained according to the best performance of networks on every participant during the training and testing process. Notably, the training process of CNN architecture has not been reproduced for Dataset1, since the average accuracies results are directly from [9]. The FBCNN code on Dataset1 and the CNN and FBCNN codes on Dataset2 are available in https://github.com/tianwangchn/SSVEP_FBCNN_12class and https://github.com/tianwangchn/SSVEP_FBCNN_40class, respectively.
For UD training method, the leave-one-session-out strategy is performed on each participant's dataset. To be specific, the network is trained and validated on the same participant's data, and 10-fold cross-validation method is used to divide the data. The sum of 1s non-overlapping segments for Dataset1 are: 648 (training) and 72 (testing), and for Dataset2 are: 1080 (training) and 120 (testing). UDmag-FBCNN and UD-mag-CNN are used to instead of the training of the methods using magnitude spectrum features as input. And, the methods using complex spectrum features are referred to as UD-comp- For UI training method, the leave-one-participant-out strategy is introduced to carry out the training and testing procedure. In UI method, a subject is leave out of the all subjects for testing process, and the remains are used to training the classifier. The total number of 1s nonoverlapping segments for Dataset1 are: 6480 (training) and 720 (testing), and for Dataset2 are: 40800 (training) and    Fig. 8(a) shows the average classification accuracies of all classification methods across 10 participants on Dataset1. Table 2  . For UD approach, the average accuracies of FBCNN methods corresponding to inputting magnitude and complex feature respectively are 3.2% and 0.86%, higher than that of CNN methods. For UI approach, the average accuracies of FBCNN methods corresponding to inputting magnitude and complex feature respectively are 3.41% and 1.86%, higher than that of CNN methods. Fig. 8  70.15±16.7%, UI-comp-FBCNN: 73.52±16.5%. As for UD approach, the average accuracies of FBCNN methods corresponding to inputting magnitude and complex feature respectively are 2.72% and 3.79%, higher than that of CNN methods. As for UI approach, the average accuracies of FBCNN methods corresponding to inputting magnitude and complex feature respectively are 3.66% and 3.37%, higher than that of CNN methods. As is shown in Fig. 9

V. DISCUSSIONS
During recent years, the deep learning-based methods have been widely applied in SSVEP classification, fully revealling the significant advantages over traditional methods in terms of classification accuracy and generalization ability [9], [21], [22], [23], [24]. However, in the design of neural networks, formerly published studies failed to take into account some prior knowledge which could help to improve the performance of a classification algorithm. Also, we verify the new network on a small dataset (Dataset1 used in this paper, 146MB) and a big dataset (Dataset2 used in this paper, 3.45GB). Fig. 11 and Fig. 12 list accuracy comparisons on the same datasets between methods in this study and previous studies. The UD training methods Combination method (92.78±10.22%) [10], IT-CCA (81.17±18.84%) [10] and the UI training methods Combined-tCCA (75±24%) [34], Compact-CNN (79±15%) [24] are introduced to compare on Dataset1 with 1s data length. The FBCCA (67±18%) [25] is cited to compare on Dataset2 with 1s data length. Compared the proposed FBCNN methods with methods [9], the average accuracies of FBCNN methods are higher than CNN between 0.86% and 3.79% on both datasets. Additionally, comparing each participant in the two datasets one by one, we find that FBCNN methods owns a distinct advantage. The average classification precisions and recalls of CNN methods and FBCNN methods vary significantly on Dataset2. Both average classification precisions and recalls of FBCNN methods are higher than CNN methods.  Especially, the recalls of all FBCNN methods are 5.7% higher than CNN methods in average. The above results prove it feasible to improve the classification performance by analyzing the correlation between harmonics. In some cases, the spontaneous EEG signals are much stronger than the SSVEP evoked EEG signals, thus the CNN using single band pay more attention on spontaneous EEG signals while neglecting the harmonics with weaker signal amplitude. We split whole single band into three sub-bands and feed them into parallel subnets of FBCNN to learn the feature of harmonics using each subnet. The contribution of the subnet to the final classification results, i.e. weight of each subnet, can be learned by full connected layers. In spite of an increase in the FLOPs of FBCNN methods comparing with CNN methods, FBCNN still remain a lightweight neural network with low computational complexity. Besides, as is shown in Table 4, we only use two band-pass filters in our experiment. Compared with three channel FBCNN, the results of two channel FBCNN are not adequate.
To intuitively visualize the difference between CNN and FBCNN, the t-Stochastic Neighborhood Embedding (t-SNE) approach is introduced to demonstrate the feature representations of CNN and FBCNN methods. The nonlinear dimensionality reduction algorithm t-SNE offers an access to visualize the high-dimensional features in lower dimensions [35]. UI-mag-CNN and UI-mag-FBCNN validated on S1 and trained on other subjects of Dataset2 are used to extract and demonstrate the patterns. Different colors of the clusters in Fig. 13, Fig. 14 and Fig. 15 stand for different SSVEP target labels. Fig. 13(a) presents the output feature clusters of the Conv2_Flatten layer (second to last layer) of UI-mag-CNN, and Fig. 13(b) shows the output feature clusters of the concatenate layer (second to last layer) of UI-mag-FBCNN. Fig.14 (a) and Fig. 14(b) illustrates the classification result clusters of UI-mag-CNN and UI-mag-FBCNN respectively. To compare the impact after SE and the pair-add-up operation, the output feature clusters of three Conv3_Dropout layers in UI-mag-FBCNN are presented in Fig. 15(a-c), and the output feature clusters of the three add layers of UI-mag-FBCNN are exhibited in Fig. 15(d-f). It can be observed that the output features become more clustered after SE and pair-add-up operation. The results acquired on both public datasets and the feature representations exhibit the effectiveness of FBCNN. The direct feature extraction and learning in CNN method [9], which rely on sole channel CNN structure from SSVEP frequency domain data without prior knowledge-based operation, may not be the most appropriate approach owing to incomplete feature extraction. More information in SSVEP data can be extracted with the help of the filter bank, and the attention mechanism enhances the distinguishability of features. The bandwidth and number of filters are set empirically in this paper, but work on the performance of FBCNN in different number and bandwidth of filters is still ongoing to further implement online SSVEP-based BCI applications.

VI. CONCLUSIONS
In summary, we propose a novel harmonic-based feature learning method for SSVEP classification, which is based on filer bank and a new CNN architecture. Two public datasets with sufficient stimulus frequencies and participants are introduced to verify the performance of all the methods for comparison. The experimental results demonstrate that UD-comp-FBCNN obtains the best performance among the compared methods and that FBCNN methods perform better than other CNN-based methods on the two datasets.