A Deep Learning Model for Remaining Useful Life Prediction of Aircraft Turbofan Engine on C-MAPSS Dataset

In the era of industry 4.0, safety, efficiency and reliability of industrial machinery is an elementary concern in trade sectors. The accurate remaining useful life (RUL) prediction of an equipment in due time allows us to effectively plan the maintenance operation and mitigate the downtime to raise the revenue of business. In the past decade, data driven based RUL prognostic methods had gained a lot of interest among the researchers. There exist various deep learning-based techniques which have been used for accurate RUL estimation. One of the widely used technique in this regard is the long short-term memory (LSTM) networks. To further improve the prediction accuracy of LSTM networks, this paper proposes a model in which effective pre-processing steps are combined with LSTM network. C-MAPSS turbofan engine degradation dataset released by NASA is used to validate the performance of the proposed model. One important factor in RUL predictions is to determine the starting point of the engine degradation. This work proposes an improved piecewise linear degradation model to determine the starting point of deterioration and assign the RUL target labels. The sensors data is pre-processed using the correlation analysis to choose only those sensors measurement which have a monotonous behavior with RUL, which is then filtered through a moving median filter. The updated RUL labels from the degradation model together with the pre-processed data are used to train a deep LSTM network. The deep neural network when combined with dimensionality reduction and piece-wise linear RUL function algorithms achieves improved performance on aircraft turbofan engine sensor dataset. We have tested our proposed model on all four sub-datasets in C-MAPSS and the results are then compared with the existing methods which utilizes the same dataset in their experimental work. It is concluded that our model yields improvement in RUL prediction and attains minimum root mean squared error and score function values.

challenging to gather run time-to-failure sensor data espe-89 cially for new machines. One way is that we can intentionally 90 run a new system upto the failure mode but it is very pro-91 longed, highly undesirable and expensive approach. Due to 92 these limitations, researchers prefer some public datasets for 93 the evaluation. In this work we have used commercial mod-94 ular aeropropulsion system simulation (C-MAPSS) dataset 95 which is basically a simulation of turbofan jet engine dataset 96 provided by NASA prognostics center of excellence [19]. 97 C-MAPSS dataset consist of four different multivariate time 98 series units with different number of engines and each engine 99 having different RUL. The dataset consists of twenty one 100 sensors with three operating conditions with respect to time 101 cycle for each engine. 102 In recent times, various work has been done on estimating 103 the RUL of turbofan engine using deep learning methods such 104 as CNN, LSTM along with their combinations and variants. 105 LSTM networks have shown better results as compared to 106 CNN based models [20], [21]. LSTM have shown excellent 107 results because they are suitable for time-series data, they 108 can learn the temporal features in multivariate system and 109 minimize the root mean square error (RMSE) with respect 110 to target predictions. In this paper, a LSTM based model 111 has been proposed for RUL prediction of a turbofan engine. 112 LSTM network can learn the association between target RUL 113 values and sensor data but it alone cannot achieve state of the 114 art performance due to various limitations like outliers, noise 115 in the sensor values, un-normalized data and un-correlated 116 sensor values. These shortcomings can reduce the perfor-117 mance of a LSTM network [22]. In this paper, we are focusing 118 on implementing some preprocessing steps on the sensor data 119 before it can be set as an input into the LSTM network. LSTM 120 network when combined with effective pre-processing steps 121 have the power to estimate the RUL with highly accuracy. 122 These added steps involve correlation analysis, data filtering, 123 normalization, and a modified piece linear degradation model 124 for determining starting point of the degradation. It has been 125 shown that the starting point of degradation which is also 126 called the initial RUL has a great impact in determining accu-127 rate RUL predictions [23]. Our proposed modified piecewise 128 linear degradation models help in efficiently calculating the 129 starting point of degradation which in combination with the 130 other pre-processing steps and LSTM network accurately 131 predicts RUL for the given engines. 132 The main contributions of our work are enumerated as 133 follows:  2) An LSTM network with effective pre-processing steps, 138 i.e. correlation analysis with data normalization and 139 moving median filter is proposed, which when aug-140 mented with the linear degradation model leads to an 141 improved RUL prediction. 142 3) Hyperparameter for the proposed prediction model 143 has been selected through iterative grid search based 144 approach [24] to further improve the accuracy of our 145 framework. 146 The organization of remaining paper is given as fol- is modelled in such a way that feature can be extracted from 220 prepared 2-D sensor data by passing raw data into convolu-221 tion layers, then flattened layer is added to convert extracted 222 2-D features into 1-D so that it can be given as an input to 223 multilayer perceptron model with dropout layer for predicting 224 RUL. Jayasinghe et al [43], proposed temporal convolution 225 in which combination of CNN-LSTM network was used for 226 turbofan engine dataset. The layers of the model have stacked 227 by first applying data augmentation to create similar type of 228 data for avoiding overfitting followed by data normalization 229 which was then followed by 1-D convolution for feature 230 extraction, lastly fully connected layer act as the bridge 231 between output of 1-D convolution layer and input of LSTM 232 layer. LSTM layer was then followed by fully connected layer 233 for output prognosis prediction. Hong et al. [44], proposed a 234 similar kind of network by stacking a 1-D convolution layer, 235 residual layer, LSTM layer and a Bi-LSTM layer. Correlation 236 analysis on sensor data for turbofan engine dataset was also 237 performed. Mo et al. [45], proposed multi-head neural net-238 work for RUL prediction of turbofan engine. This network is 239 different from the series network in such a way that they have 240 implemented the parallel branches of CNN layer in series 241 with LSTM network. Furthermore, fisher method in com-242 bination with recursive least squares and single exponential 243 smoothing was also employed to find the prediction error 244 and given it as an additional input into CNN-LSTM head for 245 optimum performance. Zhao et al. [46], proposed an adjacnet 246 neural network model for leanring the degradation pattern in a 247 sensor data. The degaradation pattern mapping learns through 248 morkov property i.e. estimating the next state of sequence 249 with the assist of only present states.

250
Many researchers have used a piecewise linear degradation 251 model in RUL prediction techniques. In this model, the start-252 ing point of the degradation is estimated often referred to as 253 the initial RUL, many authors [47], [48], [49] have chosen its 254 value on the basis of observations and no clear mechanism 255 VOLUME 10, 2022 of selecting it has been proposed. Lan et al. [23], proposed an LSTM algorithm for RUL prediction, it presented a piece      International conference on PHM [19]. This dataset was pub- The main components of turbofan engine include nozzle, 322 low pressure turbine (LPT), high pressure turbine (HPT), 323 fan, low pressure chamber (LPH) and high pressure chamber 324 (HPC). There are total of fourteen editable input parameters 325 such as fuel flow, HPC efficiency modifier, LPT efficiency 326 modifier etc. that allows you to simulate various operating 327 behaviors. C-MAPSS sensor trajectories are further divided 328 into four different units namely FD001, FD002, FD003, and 329 FD004 corresponding to different operating conditions and 330 fault modes. This dataset contains 709 engines for the training 331 and 707 engines for testing which are of same type but with 332 distinct manufacturing variation and initial wear, unknown to 333 the researcher. The description of four sub dataset units with 334 train and test trajectories and other details are given in table- 3. 335 In the start, all the engines in each sub-dataset are operating 336 normally as seen from sensor behavior and originate the 337 fault sometime later in their life cycle. In training sequence, 338 complete run-to-failure data is available with a specified RUL 339 labels as faults grows in the system and in test time degrada-340 tion values are given up to some time prior to engine failure. 341 Moreover, with different initial health conditions, there are 342 distinct number of time cycle even for the same engine in 343 dataset. The objective of this dataset is to predict remaining 344 useful life cycle of engines in each sub-unit. The actual RUL 345 label are given in the test data, which is used to validate the 346 prediction results.

347
It can be observed from

352
This paper proposes LSTM based RUL prediction model for 353 turbofan engines, which proves to be more robust than most of 354 the existing models available in the literature. The increased 355     [52], [53] to discover the relevance of 376 features with RUL. The algorithm excludes the sensor values 377 which have a very little or zero correlation with RUL, this 378 includes some parameter in engines that are basically con-379 trolled by a feedback controller and results into a near con-380 stant values or having an oscillatory behavior. These kinds 381 of parameters do not play much part in RUL predictions. So, 382 the selected feature signals are then given as an input into the 383 data filtering stage. Statistical evaluation of turbofan engine 384 degradation dataset gives us certain insight into the multivari-385 ate sensor data and furthermore reach towards the conclusion 386 that whether a considered sensor is adequate for training 387 the network or not. We can accomplish this abstraction by 388 computing the value of correlation coefficient 'r' which is a 389 relationship between the sensors and RUL labels.
where conv (x,y) is covariance between input sensor data (x) 393 and output RUL label (y),x andȳ are the mean of input sensor 394 data and RUL label, n is the number of variables in a dataset 395 and S x & S y are standard deviation of the two signals x and y 396 respectively.

397
In [44] a correlation analysis is employed for dimension-398 ality reduction to obtain the accurate results and to reduces 399 the complexity of sensor data. This technique is primarily 400 limited to FD001 sub-unit of C-MAPSS dataset, we have 401 extended this approach to entire degradation dataset and com-402 prehensively investigate the trends and irregular behaviors by 403 analyzing the correlation matrix heat map [54] of C-MAPSS 404 turbofan engine dataset. The correlation matrix heat map 405 cells show the association of three operating settings and 406 21 sensors with output RUL labels as shown in Fig. 2. The 407 correlation matrices are converted into percentage with dark 408 green color representing higher correlation as opposed to 409 light color which depict a low correlation value. The number 410 of sensors selected from each sub-unit after the correlation 411 process are given in   this correlation is more than 15% for best possible case.  The correlated data is passsed through a moving median filter 428 for removing the outliers and noises in the sensors data.

429
The choice of filter is made on its ability in removing the 430 outlier while preserving the high and low frequency contents 431 in sensor data and avoids any loss of data values. The time 432 window size of moving median filter is adaptive and vary with 433 respect to sensor values. The moving median filter belong to 434 a type of non-linear digital filter, which is used to remove 435 random unwanted noise especially when there is a high spike 436 and short-term outlier present in the data points but preserving 437 the high frequency information contents [55]. Median filter is 438 used to identify such sensor values in turbofan engine which 439 to get the filtered output after applying ascending operation.

457
So this processed data is then given to the next stages and 458 hence put a significant impact on the output.
where, y n i is the normalized value at i th time cycle for sensor 470 n, µ n is the mean value of all output of sensor n, σ n is the 471 standard deviation of all n sensor output.

D. IMPROVED PIECEWISE LINEAR DEGRADATION MODEL 473
It is observed that RUL is linear decreasing function with 474 respect to time as the efficiency of the system degrades. How-475 ever, as the system starts their operation, there is no degrada-476 tion present in the sensor readings. This pre-processing step is 477 basically implemented on output labelled data that takes input 478 from previous correlation analysis stage and employs a piece 479 wise linear degradation function on sensor values for finding 480 the initial RUL or the starting point of degradation. All the 481 labels till this deterioration point are constant out to the initial 482 RUL value while the remaining RUL lables are represented 483 as a linear line from that degrdation point up to zero life cycle 484 time.

485
In this paper, we have presented an improved version of 486 the automatic piece wise linear function [23] for output RUL 487 labeling. This approach is self-governing that is sensitive on 488 variation of the degradation trends and will automatically 489 calculate the early point of sensor deterioration. The com-490 putation of initial RUL starts by dividing entire sensor time 491 cycle with non-overlapping pattern into equal sized window 492 length of w and extract the sensor data from a given windows. 493 We then calculate the centroid of each considered window 494 by determining their mean values and geometric distance 495 calculation is performed by subtracting the two subsequent 496 windows to generate the trends in sensor data. As there are N 497 number of time cycles for given variable and window length 498 of w results into (g=N/w) geometric points for a given dataset. 499 These geometric distances are computed using Euclidean 500 distance method which is then squared and the degradation 501 pattern from g values is evidently detectable from the result-502 ing plot as shown in Fig. 3. The centroid of window w 1 is 503 first computed and subtracted from the other windows in a 504 sequence to compute the variation in sensor values to reach 505 on a point of deterioration based on the threshold value. The 506 inflection point of the curve indicates the increase in sensor 507 trends which is the initial RUL value.

508
The proposed algorithm is given as Algorithm-1 is imple-509 mented for each engine. The minimum value of initial RUL 510 among all the engines in a sub-unit is taken as the initial RUL 511 for that sub-unit. Threshold level is dependent on the rate of 512 rise in raw sensor data, its visual perception and how early 513 we need to predict the faults in the machines for maintenance 514 purpose.

515
In this paper, we have set different ranges of threshold 516 (0.01 to 0.2) and window size (5,12) for calculating the initial 517 RUL and validating the performance of our model. We have 518 used different values of window sizes in order to compute 519 the knee point in sensor data effectively. This choice stems 520  in a sequential manner with different number of hidden units 537 and a dropout layer is also added in between the LSTM 538 layers for enhancing the generalization of network to avoid 539 over fitting. It is then followed by two fully connected layer 540 with dropout layers and the final layer is the regression layer 541 as shown in Fig. 1. Fig.4 shows a basic LSTM cell that is 542 essentially consist of three control gates: input gate, forget 543 gate and output gate. The output of the cell is denoted by h t , 544 which is a short-term memory sate in a network and C t is 545 considered as a long-term cell state. The first gate in an 546 LSTM cell is forgot gate f t , which is used to unlearn selective 547 information stored in previous LSTM cell. The forget gate 548 equation is given below.
where σ ( ), is called sigmoid activation function, which can 551 control operation of forget gate. W f is the weight matrix, h t−1 552 short term state from previous cell, x t is the input of cell, and 553 b f is the bias vector of LSTM cell. The input gate controls the 554 new information entering into the cell through following two 555 equations: . This value is calculated 559 by the same short term state vector h t−1 which is further used 560 to update the new state of cell. W i and b i is the weight matrix 561 and bias vector of an input gate respectively. TheC t computed 562 from above equation is first filtered by it and then added to 563 the long term state of the cell. W c and b c are the weight matrix 564 and bias vector. After computing the value of forget gate (f t ), 565 input gate (i t ) and (C t ), long term state C t of LSTM cell is 566 updated after applying given below matrix operation where, ⊗ is basically element wise matrix multiplication 569 operation between a specified variable and C t−1 is the pre-570 vious state of LSTM cell. Finally the output of LSTM cell is 571 generated by the following two equation, The output state of LSTM cell h t , is obtained by filtering The dropout layer is added to avoid the overfitting which 581 inherently occur while training the deep neural network [58].

582
This regularization layer is added in between the fully con- 648 where e i , is the prediction error and N is the total number 649 of samples. In PHM08 data challenge competition, a score 650 function was employed to evaluate the performance of the 651 prediction model [61]. It is an asymmetrical score function 652 which means it can assign more weights to late prediction as 653 opposed to early prediction. It is described mathematically as:   four LSTM models which gave us encouraging results were 678 further tested and tuned for best performance. The models 679 along with the hyper parameters are given in table-7, LSTM 680 model structure remains same as described in the Fig.1. The 681 output of the pre-processing stages which processed sensor 682 data and RUL output labels are given as an input into the 683 LSTM network. The network is trained using the different 684 values of the hyper parameters given in the table-7 and the 685 window size and threshold ranges defined in the last section. 686 After following an iterative grid search approach, the best 687 hyper parameters are selected for the proposed model on the 688  observed by histogram distribution of prediction error in 722 Fig. 6.

723
The histogram distribution of four sub-units indicates the 724 variation of RMSE across the dataset. The x-axis indicates 725 the error or difference between predicted observation and true 726 RUL while the y-axis indicates the frequency of occurrence 727 for the given error. It can be seen from the figure that large 728 concentration for frequency of error lies in the range of 729 [-5 0] in FD001 & FD003 while for FD002 and FD004, 730 it is concentrated in [-10 0]. The data description in section 731 III shows that FD002 and FD004 has 6 operating conditions 732 with more than 200 tracking trajectories. In correlation anal-733 ysis, we have demonstrated that these two sub-units pos-734 sess irregular behavior so RUL prediction for these complex 735 sequences is a challenge for the prediction model. FD001 has 736 VOLUME 10, 2022   The robustness of our proposed model is showed by pre-752 dicting the full life cycle time of few randomly selected test 753 engines in Figures 7-10. Fig. 7 shows the actual and predicted 754 degradation results of RUL of four different types of engines 755 from a total set of 100 engines for validating the performance 756 of model for FD001. In a similar manner, Fig. 8-10 shows 757 the actual and predicted results of full cycle for other three 758 sub-units. From these prediction graphs, we analyze that the 759 predicted and actual degradation of test engine in FD001 and 760    To conclude, in what follows we reason why the pro-771 posed prediction framework is able to yield better results 772 than the existing equivalent models. The proposed frame-773 work consists of multiple stages from correlation analysis 774 to LSTM network. The significant stages in this framework 775 that greatly enhance the overall accuracy of model are cor-776 relation function to filter out irrelevant sensor variables as 777 given in Fig. 2, and the estimation of initial RUL value 778 with piecewise linear function as given in algorithm 1. This 779 function gives the starting point of degradation for sensor 780 data, and hence, we have used those values for training 781 our deep LSTM network. Moreover, the hyper-parameters performance with school and family tutoring using generative adversar-911 ial network-based deep support vector machine,'' IEEE Access,vol. 8,912 pp. 86745-86752, 2020.