Quantitative and Qualitative Analysis of Time-Series Classification Using Deep Learning

Time-series classiﬁcation is utilized in a variety of applications leading to the development of many data mining techniques for time-series analysis. Among the broad range of time-series classiﬁcation algorithms, recent studies are considering the impact of deep learning methods on time-series classiﬁcation tasks. The quantity of related publications requires a bibliometric study to explore most prominent keywords, countries, sources and research clusters. The paper conducts a bibliometric analysis on related publications in time-series classiﬁcation, adopted from Scopus database between 2010 and 2019. Through keywords co-occurrence analysis, a visual network structure of top keywords in time-series classiﬁcation research has been produced and deep learning has been introduced as the most common topic by additional inquiry of the bibliography. The paper continues by exploring the publication trends of recent deep learning approaches for time-series classiﬁcation. The annual number of publications, the productive and collaborative countries, the growth rate of sources, the most occurred keywords and the research collaborations are revealed from the bibliometric analysis within the study period. The research ﬁeld has been broken down into three main categories as different frameworks of deep neural networks, different applications in remote sensing and also in signal processing for time-series classiﬁcation tasks. The qualitative analysis highlights the categories of top citation rate papers by describing them in details. networks, long short term memory, machine learning, recurrent neural networks, deep neural networks, time-series analysis, EEG, remote sensing, dynamic time warping, transfer learning, crop classiﬁcation and multivariate time-series.


I. INTRODUCTION
A time-series is a series of observations expressed as temporal data points ordered in time, such as stocks market, human activity, audio, etc. Classification is a predictive modeling problem included in time-series data mining tasks [1]. Different classification problems are caused once dealing with rich and diverse-range time-series data domains such as, finance, medical science and engineering [2]. Variant experiments are conducted by UCR repository, the biggest set of real-world time-series data [3]. These actions along with many others are untaken in order to solve time-series classification problems [4]. Time-series classification techniques can be basically categorized to three main divisions of distance-based, using a distance function for similarity measurement, model-based, The associate editor coordinating the review of this manuscript and approving it for publication was Vicente Alarcon-Aquino . using statistical models for classification, and feature-based, using feature vectors classification methods [5].
Euclidean Distance (ED) and Dynamic Time Warping (DTW) are commonly used measurements in distance-based methods. The classification task is done by metric classifiers, such as Support Vector Machines (SVM) and K-Nearest Neighbors (KNN). SVM classifiers have been using embedded distance features for time-series data by computing EDs inside the classification task [6]. The SVM method is an effective approach in case of high dimensional data and superior in classification accuracy [7]. KNN classification is done by using the most frequent label in the k closest data points. The 1-Nearest Neighbor classifier with DTW distance (1NN-DTW) is the most effective time-series classifier, however it carries high computational cost as a concern [8]. 1NN-DTW classifier has shown a simple but strong golden baseline by achieving comparable results among other classifiers [9], [10]. Therefore, latest developments have focused on overpowering the 1NN-DTW method in time-series classification tasks [11].
Simple time-series representations, such as naïve Bayes and Hidden Markov Model (HMM), are classified using model-based methods. A naïve Bayes classifier selects the highest conditional probability among every class features [5]. HMM has also been used to classify time-series data into time-step related states probabilistically [12]. Featurebased methods, such as decision trees, Shapelet Transform (ST), and Artificial Neural Networks (ANNs), perform a feature extraction procedure to classify vector of features instead. Time-series shapelets are similar subsequences presented as features for classification [13]. Using decision tree classifiers based on time-series shapelets has led to faster and more accurate results [14]. Shapelet-based algorithms have also provided interpretable results with low performance on early time-series classification [15]. Researchers are considering ANNs for time-series manipulation mostly in modeling and forecasting [16]. Moreover, ANNs have received inconclusive results in case of complex nonlinear data classification tasks using their nonlinear learning ability [17].
Non-automatic feature selection is a challenging stage for numeric data which may lead to information loss through discretization [18]. Variant Machine Learning (ML) stateof-the-art classifiers were evaluated on both univariate and multivariate time-series datasets because of their great performance. ML algorithms are modified to more accurate timeseries classifiers, as their limited performance is probably due to the high dimensionality of large time-series datasets [19]. In recent years, deep learning has become extremely popular and widely applied to a number of classification tasks. Deep learning frameworks automatically learn and extract features from raw data, not requiring individual hand-crafted features [20]. These methods are applied to univariate as well as multivariate time-series datasets by supporting multiple inputs and outputs [21].
Deep learning methods for time-series classification can be characterized in terms of generative and discriminative models. Generative approach contains Autoencoders and Echo State Networks (ESNs), while discriminative approach contains feature engineering and end-to-end methods [11]. ESNs are used for modeling and classification of nonlinear timeseries [22]. These algorithms have been applied to different time-series data, studying the effects of ESNs network architecture on data characteristics [4]. Autoencoders are used to pre-train time-series classifiers as nonlinear feature extractors. Autoencoder deep neural networks such as Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) have been at the center of recent studies. These methods are outperforming traditional methods for time-series classification in terms of automatic deep feature generation, classification accuracy and noise tolerance [23]. Recently, a Long Short Term Memory (LSTM) RNN model has been proposed for more accurate and less complex image time-series classification, and also another trained deep RNN approach has provided useful representations for time-series classifiers [24], [25].
Nowadays, different machine learning and deep learning methods are considering time-series data collected from all sorts of domains including industry, medicine, finance, and agriculture, for variant applications such as image, video, gesture, motion, signal, agricultural, and fault classification. Due to the variety of applications, a significant number of publications are conducted in time-series classification field, noting that recent studies are mostly focused on using deep learning based algorithms for the tasks. Readers can find several comprehensive reviews and surveys in time-series data mining tasks and, more specifically, in using deep learning methods for time-series classification. The review papers have provided technical information about basic definitions, theories, and structures; existing literatures, their methods and outcomes; and current and future methods, applications and challenges [1], [5], [11], [19]. In contrast to the reviews mentioned above, a Bibliometric analysis has been carried out in this paper to visualize the trends in existing literature and overview current state-of-the-art topics to help future research.
Unlike traditional narrative literature reviews, which rely on the experience and knowledge of the researchers, bibliometric analysis provides a perspective that can easily be scaled from micro to macro levels [26], [27]. Bibliometrics is defined as measuring academic and organizational performance based on several indicators such as number of publications, number of citations, and collaboration networks [28]. Bibliometric analysis is a powerful tool for quantitatively investigating scientific outputs [29]. Bibliometric analysis of scholarly publications, lead to development of different metrics gaining insight into the intellectual structure of a broad academic discipline and evaluating the impact of scientific documents [30], [31]. A bibliometric study analyses the research productivity, top-cited publications, countries' scholarly outputs, most frequent keywords and the trend of publications to quantitatively explore a specific research area [32].
There are many Bibliometric analyses applied in different fields of studies including research performance in social science [33], green supply chain [34], knowledge management [31], university research performance [35], fuzzy research [36], drug delivery [32] and many others. Meanwhile, there are also a few Bibliometric studies close to the research area of time-series classification using deep learning, such as global research on artificial intelligence [37], analysis of global remote sensing research [38], support vector machines research trend [39], temporal topic modelling [40], analysis of affective computing researchers [41], and application of artificial intelligence to wastewater treatment [42]. However, none of these studies directly concentrates on using deep learning for time-series classification tasks. On one side, the previous published studies have not treated bibliometric approach in the field of time-series classification using deep learning to the best of our current knowledge. On the other side, bibliometric analysis alone cannot be a substitute for qualitative peer assessment [43]. Therefore, bibliometric results should be used with precautions to evaluate the relevant scholarly outputs [44]. Using qualitative analysis alongside a bibliometric study will elaborate more insight into scholarly outputs [45], [46]. Therefore, top papers for qualitative analysis were selected based on annual citation rate. Citation analysis is an important method of evaluating the impact of a research article [47]. The number of publication citations reflects its contribution to the field of interest and it is often considered as a proxy for its influential [48]. This research aims to explore the research status of deep learning frameworks used for time-series classification tasks, within the past decade, by both the bibliometric and the qualitative approaches. The study highlights the research focuses and hotspots of time-series classification algorithms and applications, that enable scholars to understand the develop trajectories, the key factors of current research, and future challenges of the research area.

II. METHODOLOGY
There are two major databases for collecting the scholarly publications data, namely Web of Science (WoS) 1 and Scopus 2 [49]. While WoS is considered one of the largest and most trustworthy databases for literature retrieval and analysis, the journal coverage of Scopus appears more comprehensive than WoS [50]. To choose a suitable database for the current research, a primary title search of ''classification'' [5] and a topic search of (''time series'' OR ''time sequence'') were run on both WoS and Scopus databases. The number of documents retrieved were 2,431 publications for WoS and 2,974 publications for Scopus database. These are the number of documents published in all years, containing the word ''classification'' in their title and the phrase ''time series'' or ''time sequence'' in their title, abstract or keywords. Hence, the data was collected from Scopus database on 6 th of April 2020 due to the larger number of publications.
In order to select the most recent documents, results were limited to the publications since the year 2010. Therefore, the number of retrieved documents have dropped from 2,974 publications to 2,309. Documents of the year 2020 are excluded from the results, and the final data consists of a total number of 2,201 documents. These are the documents published during 2010-2019 about all variant frameworks and applications for time-series classification task. The VOS Viewer software [51] is used to demonstrate annual number of publications and visualize the top 20 productive countries and the top 50 keywords in time-series classification research. VOS Viewer is a software which plots distancebased maps and clusters keywords retrieved from both titles and abstracts of research documents [52], [53]. The bibliometric visualization map includes linked nodes, where every node is a representation of a keyword, a country, an institute, or an author [54]. The total number of 2,201 documents were refined to 419 by (deep) AND (''neural network * '' OR ''learning'') search terms. These 419 publications are directly relevant to deep learning methods used in time-series classification research area.
There may be some limitations in data collection procedure of this study, as similarly in other bibliometric studies. The main dataset consists of publications available in Scopus database including classification in their title and either timeseries or time-sequence in their topic (title, abstract or keywords). Followed by that, the final dataset is refined by publications including phrases such as either deep learning or deep neural network. However, some authors may have been using the possible synonyms of these phrases instead. Although about 95% of documents are indexed by both WoS and Scopus databases [49], the study will not cover literatures specifically from WoS database. Overall, the large number of publications considered in this bibliometric analysis ensures that the literature represents the major research efforts in time-series classification using deep learning field.
The data is analysed by Bibliometrix-package, 3 which is an R-Tool for comprehensive science mapping analysis [55]. The Bibliometrix-package is designed for quantitative research in Scientometrics and Bibliometrics. The tool provides various routines for importing bibliographic data from well-known databases such as Scopus and WoS. The data collection procedure is illustrated in FIGURE 1, as Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram [56]. The bibliometric analysis results are combined with a qualitative analysis of publications content. The top 50 most frequently cited articles, among all 419 publications, are categorized into three main subjects. Followed by that, articles with a considerable number of citation per year were selected for a qualitative literature review.

III. QUANTITATIVE ANALYSIS
A total number of 2,201 papers were published on the topic of Time-Series Classification from 2010 to 2019. FIGURE 2 shows an exponential uptrend in the annual number of publications in Time-Series Classification research area. In early 2010, the yearly output of research papers was just above 100. However, the Time-Series Classification research productivity increased slowly over the first 3 years specifically in 2011, where 134 published documents were recorded. There was a very slight rise in publication rates in 2013 and 2015 with 152 and 193 papers respectively. As FIGURE 2 depicts, more and more scholars paid attention to this area from 2015 onwards, where the productivity has started to grow faster. This trend has continued to rise right through 2019, publishing about 459 documents in that year. In general, the yearly scientific production in Time-Series Classification is 4.06 fold during 2010-2019 period.     are two main research centres of this field, while Germany is in fact joining Time-Series Classification research powers in near future.
Co-occurrence analysis has been carried out among all sorts of keywords including author's keywords and Scopus keywords index. Keywords with the minimum number of 60 occurrences were included in the analysis. The bibliometric analysis in Time-Series Classification includes 12,107 keywords, in which an overall number of 50 met the threshold. FIGURE 4 shows the network structure of 50 high frequency author's keywords and keywords index. Keywords different colours indicate the initial date of their related publications, and lines show the co-occurrence links between different keywords. Time-Series Classification-related publications have been using classical Algorithms and Radiometers from the very beginning in 2014. Time-Series Classification tasks started using Learning Algorithms, followed by Machine Learning approaches in 2017.
After conducting co-occurrence analysis among keywords, it was revealed that the hottest topics in the literature are Deep Learning, Deep Neural Networks, Convolution and Convolutional Neural Network coloured in dark orange. In order to confirm the recently added keywords which may not have enough occurrences to be considered in FIGURE 4, another        Table 2 indicates, researchers are more frequently using CNN methods for classifying time-series datasets compared to Long Short Term Memory (LSTM) or other Recurrent Neural Network/s (RNN) algorithms. Interestingly, all these top author's keywords have increasing occurrences in recent publications. FIGURE 9 visually discovers the extracted keywords index from Scopus database by clustering them into variant categories using clustering techniques in combination with multidimensional scaling (MDS). The conceptual structure map of top 30 index keywords indicates that publications related to Time-Series Classification using Deep Learning can be organised into three main clusters. The red cluster, as the VOLUME 8, 2020

IV. QUALITATIVE ANALYSIS
Keywords cluster analysis helps identify research trends and status in the field of time-series classification focusing on deep learning methods. The knowledge clusters show prominent groups of themes and characterize the research directions based on the extracted keywords index from the literature. Time-series classification most productive literatures generally concentrate on novel deep learning algorithms and their application in image classification and signal processing. In this section the noticeable research clusters identified through the bibliometric analysis are qualitatively analysed. Keywords index automatic clustering is combined with a qualitative analysis of the publications content.
All documents in time-series classification using deep learning according to Scopus database are sorted by annual citation rate in descending order. Each paper's citation rate is calculated by dividing total citations received by the number of years passed since publication. Evaluating top-cited papers' content helps to identify the most significant research topics within the study areas. Publications with the highest citation rate are qualitatively analysed based on all titles and abstracts of the full paper set. The top 50 most frequently cited papers are categorized into common key research directions shown in Table 3 in Appendix. Review papers are excluded from the qualitative analysis set as they carry information about all three categories.
Documents with the highest citation rates are selected from the top of the most-cited articles list for more in-depth analysis. The nominated articles have been cited at least 10 times per year with their average citation rates ranging from 49.00 to 10.40. Top 23 most-cited articles are considered for detailed qualitative analysis since the number of citations per year suddenly drops to 8.20 afterwards. Three main research topics of time-series classification using different Deep Neural Networks (DNNs) algorithms for variant remote sensing and signal processing purposes are conducted during the analysis. The extracted research areas confirm the revealed research clusters of previous factorial analysis. The following section presents an overview of every highly cited paper purpose related to each research cluster.  (1NN-DTW) baseline. Feature extraction using Fully Conventional Networks (FCNs) has performed better than traditional approaches. Deeper neural networks have also been applied to time-series data using residual networks such as ResNet structure. The different baseline models classify timeseries from scratch with minimum complexity [9].
Time-series classification has been done by combining FCNs and LSTM from Recurrent Neural Networks (RNNs) family. LSTM-FCN has supplemented FCNs performance by raising model parameters count and minimizing dataset pre-processing. Although Attention Long Short Term Memory -FCNs (ALSTM-FCN) have provided a strong baseline using attention mechanism, the cells failed in classifying some time-series datasets [57]. Convolutional Neural Network (CNN) framework has improved time-series classification accuracy and noise tolerance by automatically generating deep input features [23]. However, several machine learning algorithms exist, outperforming the standard CNN in terms of time-series classification accuracy [2].
Computer vision classification and imputation algorithms can be used due to time-series encoding as images. Gramian Angular Fields (GAF) and Markov Transition Fields (MTF) are different frameworks used for time-series image translation. A proposed method has combined representations as GAF-MTF images and applied Tiled Convolutional Neural Networks (Tiled CNNs) for classification [58]. Using transfer learning for deep learning baseline models is counted as a replacement for time-series classification from scratch. Transfer Deep Convolutional Neural Networks (DCNNs) has improved the accuracy of different time-series classification by fine tuning the model after pre-training [59].
Different deep learning frameworks are suggested for multivariate time-series (with at least two time-dependant variables) classification. Feature learning using Multi-Channels Deep Convolutional Neural Networks (MC-DCNNs) has improved in performance compared to the traditional featurebased approaches. The baseline represents features at the last network layer as a combination of learned features from univariate time-series of every channel [18], [20]. Multivariate Convolutional Neural Network (MVCNN) is an extension of the CNN framework considering multivariate and lag-feature characteristics for multivariate time-series classification [60]. Both LSTM-FCN and ALSTM-FCN algorithms are recently developed to be used as multivariate time-series classifiers. The method has improved the accuracy and minimized the pre-processing as a result of enhancing the fully convolutional block [61].

B. APPLICATIONS OF TIME-SERIES CLASSIFICATION IN REMOTE SENSING
Deep learning-based classification frameworks have been used as the most efficient approaches for remotely sensed time-series datasets. Time-series observations from multitemporal remote sensing benefits in different vegetation types classification. Different deep learning classification algorithms are applied to process vegetation index (VI) time-series. Landsat Enhanced Vegetation Index (EVI) timeseries classify economic summer crops designed by One-Dimensional Convolutional Neural Networks (Conv1D) and LSTM. The Conv1D-based model has achieved the highest accuracy as a multi-level feature extraction. The LSTM baseline model has also been efficient in describing temporal patterns at different frequencies [62].
DNN-based approaches have integrated time-series Landsat information from USDA's Common Land Units (CLUs) for field-level crop-type classification. The baseline model has applied field-level and in-season classification on corn/soybean dominated fields with higher performance and lower costs [63]. Agricultural land-cover mapping is improving due to the speckle noise in radar images compared to optical images. Three-dimensional crops information is available using multitemporal Synthetic Aperture Radar (SAR) Sentinel-1 remote sensing data. Deep RNN-based techniques have been applied to the radar time-series data achieving great performance classification [64].
Different RNN algorithms such as LSTM model has also classified multitemporal spatial data from satellite images time-series (SITS) using alternative feature representations [65]. For a full representation of land-cover classification information for SITS, a deep learning two-branch architecture as a mixture of CNN and RNN algorithms is proposed [66]. Temporal Convolutional Neural Network (TempCNN) method has also been efficient in more accurate classification of SITS by learning temporal and spectral features automatically [67].

C. APPLICATIONS OF TIME-SERIES CLASSIFICATION IN SIGNAL PROCESSING
Deep learning has proposed variant signal classification methods using visual inspection of signals. Many sleep disorders have been recognised via sleep stage (every 30 seconds of signal) classification. DNN methods based on linear spatial filtering and temporal convolutions have exploited the multivariate and multimodal time-series of Polysomnography (PSG) signals containing electroencephalography (EEG), electrocardiography (ECG), electrooculography (EOG), and electromyography (EMG). The model classifies real-time sleep stages performing with minimum computational time and cost compared to convolutional networks or decision trees [68]. Heart interbeat RR time-series are composed into Intrinsic Mode Functions (IMFs) for feature evaluation. DNNs are coupled with deep features extracted from RR time-series and EEG signals for classification of sleep stages [69].
Different deep learning methods are developed for nonstationary time-series classification as EEG signals in motor imagery classification. Multilevel convolutional feature extraction methods have been applied to CNNs for EEG decoding [70]. Restricted Boltzmann Machine (RBM) based deep learning scheme has improved the performance of motor imagery classification using an additional network output     layer of Frequential Deep Belief Network (FDBN). In FDBN, frequency domain input of EEG signals is generated by either Fast Fourier Transform (FFT) or Wavelet Package Decomposition (WPD) and passes through RBMs for classification [71]. Semi-supervised paradigm of Deep Belief Networks (DBNs) has provided fast classification of multiple timeseries EEG waveforms. DBNs method has also resulted powerful anomaly measurement of clinical EEG signals using raw time-series inputs [72].
Different deep learning algorithms are proposed for multivariate time-series classification using multivariate sensor signals from semiconductor manufacturing process. Advanced Fault Detection and Classification (FDC) along with CNN model (FDC-CNN) has outperformed others by locating time-series information which represent process faults [73]. Similar deep learning approach of DCNN has been applied to manufacturing gearbox fault type and fault severity classification. Underlying deep features of timefrequency images have been extracted for classification. Graphical images are encoded from one-dimensional timeseries vibration signals through wavelet analysis. The image processing advantage of DCNNs has resulted great classification accuracy in dynamical systems [74].
The study provides a detailed quantitative and qualitative analysis, based on publications in time-series classification retrieved from Scopus database from 2010 to 2019. The bibliometric analysis on time-series classification discovered the most recent subfield of study as deep learning. The dataset is refined by deep learning exploring the development status in terms of publication years, countries, sources, keywords and topics. The annual publications in time-series classification using deep learning research field has increased in the past 10 years, showing the potential to grow rapidly in the future. The analysis of international cooperation helps fresh researchers to spot potential research collaborators. According to countries related publications, China is the most productive country collaborating with over a hundred publications in time-series classification using deep learning during 2010-2019. The USA is also known as the most collaborative country in this bibliographic set, as American corresponding authors have several collaborations with other nations.
Studying the evolution of journals publishing time-series classification using deep learning research, identifies highlytargeted sources for new researchers of the field. As a result of the source growth analysis, IEEE Access appears to be the highly productive platform in which most researchers on time-series classification using deep learning publish their papers. Finding research hotspots is possible by calculating the total appearances of keywords during this 10-year period. The analysis of author's keywords occurrence shows that scientific productions in time-series classification using deep learning are mainly concerned with convolutional neural 90212 VOLUME 8, 2020

V. CONCLUSION AND FUTURE WORK
networks, long short term memory, machine learning, recurrent neural networks, deep neural networks, time-series analysis, EEG, remote sensing, dynamic time warping, transfer learning, crop classification and multivariate time-series.
Topical analysis within the field of time-series classification using deep learning provides the active core categories in this area. The factorial analysis of keywords index and the qualitative analysis of top-cited publications content demonstrate the same research themes concentrated on deep learning algorithms, remote sensing and signal processing applications. A large number of publications in time-series classification using deep learning are directing their studies to the different baselines of deep neural networks with special emphasis on deep convolutional neural network frameworks.
This paper is intended to explain how the research frontiers in time-series classification using deep learning area have changed over the past decade. The paper is crucial for understanding the usage of deep learning algorithms for time-series classification tasks and following the research directions for further developments in the field. Research on improving deep learning algorithms and extending their applications for time-series classification tasks are still ongoing. It can be predicted that a wide range of researchers will start working on the mentioned hot research topics of time-series classification research field focusing on deep learning methods, leading to a great number of publications. Table 3 shows the most cited papers which are categorized into three prominent research subjects.