An interpretable deep learning classifier for epileptic seizure prediction using EEG data

Deep learning has served pattern classification in many applications, with a performance which often well exceeds that of other machine learning paradigms. Yet, in general, deep learning has used computational architectures built, albeit partially, by ad hoc means, and its classification decisions are not necessarily interpretable in terms of knowledge relevant to the application it serves. This is often referred to as the black box problem, which in certain applications, such as epileptic seizure prediction, can be a serious impediment. The purpose of this study is to investigate an interpretable deep learning classifier for epileptic EEG-driven seizure prediction. This neural network is interpretable because its layers can be visualized and interpreted as a result of a novel architecture where the learned weights follow from signal processing computations such as frequency sub-band and spatial filters. Consequently, the extracted features are no longer abstract as they correspond to the features commonly used for decoding EEG data. In addition, the network uses layer-wise relevance propagation to reveal pertinent features which can further explain the computations leading to the decisions. In seizure prediction experiments using the CHB-MIT data set, the method produced classification results which improved on the state-of-the art, with first network layer filters corresponding to clinically relevant frequency bands, and the input channels in the brain location in which the seizure originates contributing most significantly to the network predictions.


I. INTRODUCTION
D EEP neural networks have extended considerably the ability of common neural networks to learn and classify patterns, with striking, unprecedented results in long standing applications, and in challenging new ones as well [1]. However, in many other important applications, such as EEG signal classification for epileptic seizure prediction, which is the subject of this study, pattern feature learning in deep neural networks, or deep learning (DL) [2], suffers from what is often referred to as the black box problem, where, in general, some prevalent network architecture is used, without explicit justification, to have its parameters learned from data by often adhoc trial and error experimentation. As a result, better classification is often missed. Moreover, even when the classification is accurate, the results come essentially with no interpretation of how the network reached its classification decisions. The ability to interpret these decisions may not be an issue in simple classification tasks where a wrong outcome is of little consequence. However, in general, interpretation aids in learning better network parameters, for instance by biasing the network structure to favor learning of applicationrelevant features. In domains such as healthcare, it may be essential to develop efficient applications relevant in clinical settings. In this study, we consider a neural network to be interpretable at two levels: First, by designing layers that bias learned filters toward common signal processing computations, such as frequency sub-band and spatial filtering, which are relevant to the application that the network serves and, second by explaining the influence of the various input variables, or of the network learned features, in reaching the classification decisions. For instance, the deep neural network we investigate in this study for epileptic seizure VOLUME 4, 2016 prediction is interpretable in that it uses a convolution layer similar to a filter bank to extract characteristic filters corresponding to clinically relevant frequency bands, and the input channels in the brain location in which the seizure originates contribute most significantly to the network predictions.
The ability to interpret classifiers has been of general interest in Artificial intelligence and has first appeared in symbolic reasoning which supports decision making in expert systems, such as MYCIN [5] which sets to diagnose patients on the basis of reported symptoms and medical tests results, and also GUIDON [6] for knowledge based tutoring, and SOPHIE [7] for hazard identification. In machine learning as well, being able to explain the relationship between the input and output of predictive models, and interpret the outcomes of this relationship, has been a concern [8]. Notable contributions in this regard are decision trees and related classifying structures, considered interpretable from the flow of their successive decision making stages [9].
Interpretable neural networks are of relatively recent interest [11], although the need for these is quite plain because, as mentioned earlier, neural networks, particularly deep neural networks, are currently used mostly in a black box manner, to process a multitude of classification applications which would be more accurate and of more flexible and general usage were their internal architecture are interpretable. Noteworthy investigations include studies and applications of saliency maps to visualize and understand non-linearity in neural networks [10], and computer vision studies which related neural activity to image filtering [12]. The investigation in [12] was able to determine that the first-layer filters learned by a deep belief network for natural image patch recognition is analogous to location, orientation, and spatial frequency filters like Gabor filters used for edge detection. For digits recognition using the MNIST dataset, [13] observed that the network learned low-level features similar to those found in stroke detectors typically used for text localization. For accrued interpretation abiliy [14], an explanation producing model can be construed using an architecture designed so as to simplify interpretations of internal representations and corresponding processing.
A means to explain a deep neural network computations is the layer-wise relevance propagation (LRP) scheme [15], by which a network decision is decomposed into relevance scores for each neuron, starting from last layer and propagating back towards the input. LRP has been a useful explanation tool in many applications. For instance, [16] used LRP to explain digit recognition and gender classification using the AudioMNIST dataset, which contained spoken digit records. The spectrogram representation showed that different areas of the input were critical to each class for digit classification, and the low frequency range was a determinant for gender classification. Moreover, based on waveform data, large magnitude data were determined also important. In another study [17] LRP was used to explain the classification of subjects EEG recordings while imagining left and right hand movements. The relevance score was calculated for each channel at each time point. The channel relevance of a time point reveals a typical lateralized motor activation pattern, which, when averaged over all epochs, yields a similar pattern.
In spite of the evident progress in building interpretable pattern classifiers for several important image-based applications, the subject remains actual and challenging for waveform data. Little work has been done in EEG-based applications, and none in epilepsy seizure prediction, the subject of this study. This study takes up the problem of developing an interpretable deep learning neural network applicable to epileptic seizure prediction in EEG recordings. Epilepsy seizure prediction is a subject worthy of investigation because epilepsy affects about 2.4 million people of all ages worldwide each year [18] and involve seizures with the risk of periodic disruptions in cognitive and behavioral functions. Predicting seizures would obviously benefits patients significantly, and also lighten physicians workload. Electroencephalography (EEG), which involves recording brain activity with electrodes placed on the scalp, has proven to be a reliable non-invasive clinical approach for epilepsy diagnosis. Predicting the eventual occurrence of seizures relies on identifying the pre-ictal period prior to the onset of a seizure [19], during which EEG recordings show different patterns from the patterns of the seizure and also from the earlier periods, so-called inter-ictal periods. As a result, the classification of inter-ictal and pre-ictal states simplifies the prediction of seizures. This study provides three contributions to the field: 1. Design of an architecture following the filter bank common spatial pattern (FBCSP) paradigm and build an explanation-producing model that biases learned filters toward relevant common sub-band frequency and spatial filters, 2. interpretation of the network abstract features encoding, by learned filters visualization and, 3. explanation of the network model decisions by layer relevance propagation. We tested the model on the CHB-MIT dataset for epilepsy prediction and its results outperformed those of the current state of the art. The model architecture showed a fair interpretability. Indeed, we found that the first layer trained filters gather data from specific frequency bands . Explanation of the model's decisions for several trials of patients with focal seizures reveals that the input channels in the brain location from which the seizure originates contribute most to the model's prediction.
The remainder of this paper is organized as follows: Section II describes the data set and the proposed architecture; Section III details the experimental results, and Section IV contains a discussion.

II. MATERIALS AND METHODS
The functional diagram of the seizure prediction task is illustrated in Figure 1. The proposed framework consists of three main steps: the first step consists of pre-processing and segmentation of the data(Section II-A). This is followed by training and evaluation of the neural network (Section II-B). The resulting models are interpreted by visualizing the learned filters and explaining the model decision for several trials (Section III).

A. DATASET DESCRIPTION AND PRE-PROCESSING
The dataset used for this study is the publicly available CHB-MIT dataset collected at the Boston Children's Hospital [20]. CHB-MIT contains 940 hours of long-term continuous multichannel scalp EEG recordings collected from 23 pediatric patients aged 1.5 to 19 years as shown in Table 1. A minimum of 17 electrodes was used in all trials distributed according to the international standard 10/20 system. The sampling rate was set to 256Hz. Using a notch filter with an upper cutoff frequency of 50Hz and a band-pass filter with a bandwidth of 0.5-70Hz, we eliminated noise and artifacts and focused on relevant frequencies. Based on published literature, we set the pre-ictal period to be 30 minutes before the onset of the seizure, as outlined in [21], [22], and eliminated 30 minutes after the end of the seizure to exclude effects of the post-ictal periods. Subsequently, we divided the recordings into nonoverlapping 5-second-windows yielding 529,415 and 66,782 samples of inter-ictal and pre-ictal activity respectively.

B. NEURAL NETWORK ARCHITECTURE
The deep neural network architecture uses the Filter Bank Common Spatial Pattern (FBCSP) algorithm [23] as follows. FBCSP aims at finding spatial filters that map the raw data into additive components that are capable of discriminating between the sources more efficiently. FBSCP is widely used for decoding EEG data in different applications, such as brain-computer interfaces experiments [24], mental workload estimation [25], major depression detection [26], and epilepsy prediction [27], [28]. The algorithm is composed of two main components: (1) A filter bank and (2) Spatial filtering using the Common Spatial Pattern (CSP) algorithm.
• The Filter bank consists of a set of band-pass filters that separates the input signal into multiple signals, each corresponding to a unique frequency sub-band of the original input. • The spatial filters are linear transformations which project raw channel data into a spatial space known as "source space" to separate sources of activity [29]. The CSP algorithm computes a transformation matrix which maximize the variance of the output signal for one class and minimize it for the other. The outputs of the algorithm are generally used to extract features such as the log-variance of each sub-component in all sub-frequency bands. These features are then used for the classification task.
In this study, we followed the steps of this algorithm to design a neural network architecture that simplifies the interpretation of its layers. The use of FBSCP-inspired architectures for different applications has been very little studied before. The study by [24] suggested a convolutional neural network (CNN) with convolutional layers similar to the bandpass and spatial filters used to decode and visualize task-related information from EEG recordings. An alternative similar compact architecture [17] has been used to classify EEG signals from different brain-computer interface paradigms. Neither of these studies considered long, continuous EEG data. Instead, they used the simpler data of event-related potentials (ERPs), which are brain responses to a specific sensory, cognitive, or motor events. [31] have investigated a convolutional neural network architecture whose first layer function is equivalent to spatial filtering for sleep-stage classification using multivariate, multimodal continuous timeseries data. However, these studies did not include classification decisions explanations or interpretations. In this paper, we present an architecture that is designed to take into account the type of long continuous EEG data. Moreover, a number of interpretations and explanations have been provided to delve further into the architecture. Figure 2 and Table 2 show the overall diagram and the full detailed description of the proposed architecture.
• The network first layer performs a standard temporal 2D convolution that learns a set of band-pass filters to output multiple components, each representing a frequency band within the original signal. This step is equivalent to the filter bank stage of the FBCSP algorithm. With a temporal kernel that is half the sampling frequency, it is possible to capture frequency information starting at 2 Hz. Following this operation, the batch normalization is used to stabilize the training. • The subsequent component of the architecture starts with a depth-wise convolution, which is the application of convolution filters to each feature map (output from the previous layer) independently from the other maps.
To learn spatial filters, this type of convolution is implemented using kernels of shape (C, 1) where C is the number of channels. Subsequently, batch normalization, non-linear activation and average pooling were consecutively applied. A dropout layer is added to regularize the model. • For the feature extraction from the activity source signals learned in the previous layers, we applied a combination of 2D convolutional layers, a nonlinear activation layer and an averaging layer. The outputs are finally passed through a dropout layer. • The last stage of the architecture is a fully connected layer that flattens the features into a one-dimensional vector that is fed to a Softmax classifier.

2D Convolution Classification
Dense Input FIGURE 2. Diagram of the proposed architecture. The network processes EEG inputs with standard convolution with filters of shape (1,128) allowing learning frequency filters. It uses depth-wise convolution to learn spatial filters for each feature-map output of the previous layer separately. Finally 2D convolution is used to extract features. The outputs of the pipeline are finally fed to a fully connected layer.

1) Filter visualization
In neural network terminology, the learned filters are simply the weights of the convolutional kernels of the network. Visualizing the learned filters allows us to see how each layer extracts information from the input. Generally, for standard convolution layers, interpreting the filters proves challenging since it performs both an in-channel and in-space computation at the same time. Using the 2D convolutional filter of size (1, 128) and the depth-wise convolution to learn filters for each input channel separately, it is possible to interpret time convolution as band-pass frequency filters and depth convolution as spatial filters.

2) Layer-wise relevance propagation
The RLP technique explains neural network decisions by assigning a score to each input point (e.g., pixels) related to its relevance to the classification decision. RLP is based on back-propagating the prediction score f (x) according to specific propagation rules (see Figure 3). The prediction score is redistributed from the output layer down to the neurons of the lower layer and so forth until it reaches the input layer. For each point in the input layer, the relevance   score corresponds to its contribution to the decision. A high relevance score indicates a relevant pattern. On the other hand, parts of input with a low relevance score are considered irrelevant. Let f (x) and R (l) j be the prediction score and relevance score of the neuron j in layer l respectively. Any propagation rule satisfying the following properties could be used for PRL. First, the relevance must be preserved between layers, so that the following equation is verified.
Moreover, the relevance score of a node should be equal to the sum of the relevance score coming from nodes in upper layers and redistributed in same amount to nodes in lower layers as indicated in Equation 2 .
where R (l,l+1) j←k is the relevance score sent from the neuron k in layer l to the neuron i in the next layer l + 1. Finally, the propagation rule must ensure that the relevance scores are related to the neuron activation or inhibition; a positive score corresponds to the existence of a feature whereas a negative or null score indicates to the absence of a pattern. There are several propagation rules that have proven effective in practice that satisfy the constraints listed above. Basic rule (LRP − 0) : This intuitive rule redistributes the relevance score in proportion to the contribution of each input to the neuron's pre-activation.
where a j is the neuron activation from the previous layer and w jk is the weight of the connection from unit j to unit k. Epsilon rule (LRP − ϵ) : To ensure that R j does not take unbounded values for small or null values of neuron activation, a positive term ϵ is added to the denominator.
where a j is the neuron activation from the previous layer and w jk is the weight of the connection from unit j to unit k. Gamma rule (LRP − γ) : The LRP − γ rule denoted by the equation 5 is used to highlight the positive contributions over the negative contributions. The parameter γ controls the importance of positive evidence.
where a j the neuron activation from the previous layer, w jk is the weight of the connection from unit j to unit k and w + jk is the positive part of the weight. As suggested in [32] we used LRP − 0 rule for the upper layers, the Epsilon rule for the middle layer and the Gamma rule for the lower layers.

D. CLASSIFICATION AND IMPLEMENTATION DETAILS
As mentioned earlier, an inter-ictal and pre-ictal segments classification could simplify seizure prediction. However, epileptic patterns vary widely from seizure to seizure as well as from patient to patient, which makes binary classification challenging. Seizure prediction can be performed through general cross-subject models applicable to all patients or VOLUME 4, 2016 by patient-specific modeling applicable to each patient individually. Models that are patient-specific are generally impractical since it requires recording a sufficient number of seizures for each patient. Cross-subject modeling does not require treating each patient separately, but it faces the major challenge of adapting the prediction algorithm to unseen data from new patients, mainly due to the high variability of crosssubject EEG patterns [33].
In this study, we focused mostly on the patient-specific modelling to evaluate the proposed architecture. Accordingly, a single architecture was designed and trained for each subject separately. Pytorch [34] was used to implement the proposed architecture. For data pre-processing The MNE-Python package [35] was utilized. To ensure reliable generalization performance, a 5-fold stratified cross-validation test setup was used . A holdout validation is nested within a crossvalidation procedure in order to further divide the training set of each fold into a validation set and a training set so that the early stopping criteria can be enforced to prevent overfitting. In fact, the training runs up to 500 epochs, or until the validation loss remains constant for at least 20 epochs. Across all tasks, we use the gradient-based ADAM optimizer with coefficients β 1 , and β 2 of 0.9 and 0.999 respectively because it is fast and reliable for reaching a global minimum. We use a learning rate of 0.005 and a dropout regularization value of 0.25.

III. RESULTS
In the following, we applied the proposed architecture to the classification of inter-ictal and pre-ictal brain states for seizure prediction. After the model training, the next step is to visualize the learning filters and explain the decisions made by the neural network using LRP-based technique.

A. PATIENT-SPECIFIC SEIZURE PREDICTION
Using the CHB-MIT data, we evaluated the proposed architecture for specific-subject seizure prediction on 23 patients. Table 3 shows some performance measures such as prediction accuracy, sensitivity, specificity, precision, F1-score, Area under the ROC Curve(AUC) and false alarm per hour for each patient model. Across all patients, the overall averaged accuracy, sensitivity, specificity and F1-score across all patients are 90.9%, 96.1%, 84.6%, and 91.9%, respectively. An averaged area under the ROC curve of 0.918% was achieved. The models have a reasonably low averaged false prediction rate per hour(FPR/h) of 0.041 indicating good predictive power.
For further evaluation of the proposed architecture, we compared our results to earlier publications that employed the same dataset, as given in Table 4. In the previous works considered, CNN architectures with different numbers of layers were used, such as the single-layer architecture as in [36], the three-layer architecture as in [37] and the five-layers architecture as in [38]. The authors of [28] used the FBCSP algorithm followed by a CNN classifier. The proposed archi-tecture achieved the highest sensitivity with the lowest false alarm rate.

B. FILTER VISUALIZATION
Following model training, the interpretation of the learned data representation was conducted. As previously stated, the proposed architecture's first layer is supposed to be equivalent to a filter-bank. As a result, we are especially interested in seeing if the model was able to learn band-pass frequency filters. Therefore, for all subjects we visualized the filters learned on the first layer of the various patient-specific models. The convolution filters for the first layer of patient 20's model, as well as the frequency domain representation derived using the Fast Fourier Transform (FFT) are shown in Figure 4. The frequency bandwidths were calculated using the FFT. Figure 5 shows the frequency bandwidths of the seven learnt first layer filters in each of the 23 patients' subject-specific models.

C. EXPLAINING MODEL DECISION
The LRP technique is used to explain the model decision for many samples, which is the third level of interpretability explored in this work. As outlined in Section II, LRP computes, on a sample basis, the relevance scores for individual features related to their contribution to the ultimate classification decision. Positive relevance values suggest features that support the classification decision, whilst negative values indicate features that are irrelevant to the prediction. The relevance scores of individual features for successfully and inaccurately detected pre-ictal EEG samples were determined in this study. To display the topographic map, the relevance scores were averaged across time. Figure 6 shows the topographic representation of the relevance scores for various pre-ictal samples from seven patients with focal frontal seizures.

D. CROSS-SUBJECT SEIZURE PREDICTION
To evaluate the patient-independent model we used all of the data from the 23 patients at CHB-MIT. Thus, we divided the dataset into three stratified sets with the same proportions of classes : the training set, the validation set, and the test set. The proposed architecture yields satisfactory results. With a false prediction rate of 0.6/h, we were able to achieve a sensitivity of 67.17%. An F1 score of 65.84% was achieved.

IV. DISCUSSION AND CONCLUSION
This study investigated an interpretable deep learning model for seizure prediction using EEG signals. Its evaluation was conducted in three steps.
As a first step, we created an interpretable deep learning architecture whose earlier layers act according to the FBSCP scheme. The architecture was tested with a patient-specific seizure prediction task using the CHB-MIT dataset. The proposed architecture achieved a reasonably high level of prediction accuracy. Table 4 shows the benchmark of recent seizure prediction methods. Because these methods have been evaluated according to different metrics, the proposed  classifier has been evaluated using several metrics commonly used in seizure prediction. From a clinical perspective, it is desirable to have a high sensitivity and a low false alarm rate. Authors of [37] proposed a three-layer CNN architecture VOLUME 4, 2016  that yielded a sensitivity of 81.% and FPR of 0.16/h as tested with 13 patients from the CHB-MIT dataset. The study in [36] adopted a more compact single-layer CNN which performs much better, giving a better sensitivity of 94.2% a high accuracy of 95.6%, and a specificity of 96.9%. However, the evaluation is incomplete because no false alarm rate was reported. Another patient-specific CNN classifier has been described in [28]. The FBCSP algorithm was applied prior to the feature extraction step. The authors reported accuracy, specificity and sensitivity values of 90%, 92%, and 92%, respectively, with a relatively low false alarm rate of 0.12/h. A more advanced approach [37] used a five-layer one-dimensional binary convolutional neural network. They Our next focus was on the interpretation of the learned filters. We found that the first layer's filters were found to be similar to band-pass frequency filters and moreover, each patient-specific model has its own set of filters. Additionally, similar filters, such as those in ranges 0-5, 5-20, and 40-45 appear frequently in almost all subjects as shown in Figure 5. The model learns low-and high-frequency filters in the range 0 to 60 Hz range for each subject, which is critical for the epilepsy prediction task, since abnormal seizure discharge is primarily observed in the 5 to 50 Hz frequency range. Likewise, we found that, the models recover essential features of each patient by learning filters with a frequency of 25 Hz or higher, which is consistent with the fact that epilepsy prediction relies more on characteristics in the gamma band  that are more relevant than other bands for epilepsy prediction [30].
Finally, LRP enabled us to interpret several of the classification decisions. We found that features extracted from the channels in the region of the seizure origin were shown to be the most relevant features for pre-ictal segments classification. Hence, we determined that well classified samples with a high prediction value (Figure 6 top row) possess high relevance scores in the frontal regions where the seizure will occur, while misclassified samples (6 bottom row) displayed a distribution of relevance scores more broadly distributed throughout the scalp.
Since EEG data vary greatly between subjects and only a few patients are available, developing patient-independent models is a complex task. Therefore, most researchers simplified the problem to develop models that are patientspecific. To our knowledge, this is the first study to examine between-subjects modeling. The proposed architecture yields satisfactory results when tested on the entire dataset of the 23 patients of the CHB-MIT dataset. However, due to the substantial variability of EEG data between patients, crosssubject seizure prediction performed somewhat worse than patient-specific modelling, but the model appears to have potential applicability to data from unknown subjects.
In summary, we introduced a novel interpretable neural network architecture to simplify its opaque representation of data. The proposed architecture is based on the common FBCSP paradigm where its layers where correspond to known signal processing calculations, such as sub-frequency band and spatial filtering. The architecture performance was evaluated using the CHB-MIT dataset for the patient-specific prediction task. The proposed architecture has achieved a reasonably high predictive accuracy compared to other deep learning methods. Next, the model was interpreted by visualizing the learned filters, showing that the first-layer filters are similar to the band-pass filters. Finally, using the LRP, we were able to explain several model decisions. We observed that for the pre-ictal segments, the channels in the seizure origin region were the most relevant characteristics for classification.
The study could be strengthened by using larger amounts of data and using different EEG databases. Furthermore, it is highly useful to study how to transfer learning for crosspatient modelling, which could help to learn new representations shared between-data subjects that would transfer knowledge gained from multiple patients to new unseen patients. Finally, the proposed architecture and explanation scheme can be applied to other EEG-based classification tasks, such as seizure diagnosis and seizure type categorization, as well as autism and Alzheimer's disease detection.
IMENE JEMAL received the degree in computer science engineering from the National School of Engineering of Sfax (Enis). In 2017, she joined the Computer Science Laboratory of the University of Pau and Pays de l'Adour, as a research intern. She is currently pursuing a Ph.D. in telecommunication at the Institut National de Recherche Scientifique-Centre Eénergie Mateériaux et Teéleécommunications de Montreéal. Her research interests include pattern recognition, machine learning, deep learning and explainable AI and their application to biomedical data.
NEILA MEZGHANI received the degree in telecommunications engineering from the Higher School of Telecommunications of Tunis (Sup'Com), the master's degree in information technology from the National School of Engineers of Tunis, and the Ph.D. degree from the Institut National de Recherche Scientifique Centre Énergie Matériaux et Télécommunications, Montréal. She is currently a Data Scientist Professor with Université TELUQ (Quebec University) and a Researcher with the Centre de recherche du Centre hospitalier de l'Université de Montréal (CR-CHUM). She is the author of two patents and about 100 peer-reviewed publications in renowned scientific journals and international conferences. Her research interests include biomedical data mining and classification, artificial intelligence, decision support systems in the medical Field, and mobile health. She has the Canada Research Chair in biomedical data mining.