Defect Depth Determination in Laser Infrared Thermography Based on LSTM-RNN

Carbon fiber reinforced polymer (CFRP) has been increasingly used in aviation industry since it significantly enhances the performance of aircraft. However, imperfections inside the CFRP structures pose a threat to aviation safety. Apart from the defect shape and size, flaw depth is crucial to assess the defect severity. In this work, we utilize a laser infrared thermography (LIT) system to inspect an aviation CFRP sheet and adopt a long-short term memory recurrent neural network (LSTM-RNN) to determine the defect depth. Thermographic sequences obtained by LIT are processed using thermographic signal reconstructions (TSR) method. Raw data and TSR processed data are separately used to train and test the LSTM-RNN. Results show that background noises in the original thermal signals can be effectively reduced by the TSR method, which is helpful for the models to learn the signal characteristics. Compared with two traditional methods, recurrent neural network (RNN) and convolutional neural network (CNN), we find the LSTM-RNN outperforms.


I. INTRODUCTION
Active infrared thermography (AIRT) is a non-contact, wide range, and rapid technique [1]- [4], which has performed effectively in defects detection of carbon fiber reinforced polymer (CFRP) components. The front surface of inspected structure is excited by an external heat source. Internal imperfections of less effusivity appears as an area of higher temperature than surrounding sound materials during the stimulation course. This feature can be captured by a thermal camera, making defects detection possible. Traditional excitation sources include microwave [5], ultrasound [6], and flash lamp [7], each of which has advantages and disadvantages due to the different physical principles. Recently, with the development of laser technology, laser infrared thermography (LIT) has attracted increasing attentions. Li et al. proposed a new surface crack imaging method by using laser as thermal excitation source [8]. Qiu et al. adopted a laser to heat metal surface and studied the relationship between surface crack size and temperature distribution [9]. Wang et al. utilized a laser to heat aerospace composite plates in order to detect internal defects [10]. Swiderski  internal delamination defects of carbon fiber composite materials by using a laser as thermal excitation [11]. Compared with traditional thermal excitation source, laser has the advantages of higher energy density, higher accuracy and great controllability. The laser excitation energy can be controlled by adjusting the parameters of the laser pulse waveform, frequency and intensity through a computer. Moreover, the radius of laser spot can be controlled by optical lens system, thereby changing the heating area of the specimen. Therefore, LIT is a flexible non-destructive technology (NDT), which can accurately quantify the heat absorbed by the surface of the specimen. However, restricted by resolution limit and by the sensitivity of infrared cameras, it is difficult to find out defects with small aspect ratio or beneath the surface. To improve image contrast and signal-to-noise ratio (SNR), thermographic signal reconstruction (TSR) was adopted to process thermal response signal [12], [13].
Recently, neural networks (NN) have prevailed in pattern recognition [14], automatic control [15], signal processing [16] NNs have also advanced the defect classification and depth determination with infrared thermography NDT [17]- [21]. The featured parameter of the defects can be extracted from NNs trained with data obtained through simulations using finite element method (FEM) or VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ through experiments. Due to its particular structure, long short-term memory recurrent neural network (LSTM-RNN) is specialized in processing time sequence information, and has been applied in the natural language processing [22] and data prediction [23]. While thermographic sequences recorded by infrared camera could be regarded as temperature-time sequences of each point on the surface of inspected structure, where internal defects at different depths exhibited different characteristics in their sequences. Hu et al. used thermal contrasts over time in the cooling process as input of LSTM-RNN, which classified the debonding, adhesive, pooling, and liquid ingress defects in honeycombstructured materials. Result showed that the LSTM-RNN model is capable of defects classification [24].
In this work, we focused on the depth determination of embedding defects in CFRP laminates. A LSTM-RNN model designed for processing temporal sequences will be constructed. LIT inspected an CFRP specimen and thermal responses (temperature evolutions) in after-exciting period (cooling phase) will be used as data sets. TSR method will be utilized to reconstruct the thermal signals as another data sets. Temperature evaluation sequences from two data sets will be used to train and test the model respectively. Results will show that the LSTM-RNN model trained by data sets from TSR method has great accuracy in determining the depth of defects inside CFRP laminates. Compared with two other methods, we will show that the LSTM-RNN method is appealing in infrared thermal sequence processing.

A. SPECIMEN PREPARATION
Three pieces of carbon fiber unidirectional prepreg composite laminates were used as experimental specimens, numbered C1, C2, C3 (Fig. 1). The size of each specimen is 180mm×180mm, the thickness of the specimen skin is 3mm, the single-layer prepreg thickness is 0.125mm, and the adhesive film thickness is 0.2mm. The debonding defects was simulated by embedding Teflon films between layers, since Teflon and air have similar thermal properties. Since the initial specimen design plan was not made for this experiment, the defect size of the three specimens parts were not the same. Defects at shallow depths have small sizes, while those at deep depths have large size.

B. EXPERIMENTAL SETUP
A self-built LIT system illustrated by Fig. 2 was used in the experiment. The system consisted of three main parts: laser transmitter, infrared camera and computer. A pulsed laser with center wavelength of 1064mm was used to excite surface of specimen under test. The incident laser diameter is 4mm, and the pulse laser width range is 1ms∼500s. Due to the wide distribution of the embedded defects, concave lens, beam splitter and beam expander (from left to right) were used to diffuse the laser. A Focal Plane Array (FPA) infrared camera (from FLIR, 7.7-9.3µm) with a  320 × 240 pixels array was adopted to record the surface temperature, forming a thermographic sequence, as shown in Fig,3. A mini USB oscilloscope (Analog Discovery 2, AD2) was used to control the electrical signal of the laser transmitter to regulate the parameters of the laser.
Experiments were carried out twice for each of the three specimens, using the same experiment settings. The distance between the sample and the front concave lens was 500mm, the distances between the three lenses was 50mm, and the distance between the beam expander and the laser was 5mm. The detail of laser settings was shown in Table. 1.The diameter of the laser spot on the surface was measured to be 300mm, and the center of spot was adjusted to locate between the middle two rows of defects. From the beginning of laser excitation, the infrared camera started to record temperature distribution of the specimen. The acquisition frequency was 50Hz and the acquisition time was 20s. 1000 thermal images were recorded  for each experiment. Analyzing the thermal image obtained at the time of 10.0s, the red squares in Fig. 4 show the observable defects.

III. DEPTH DETERMINATION METHOD A. LSTM-RNN
Compared with other networks, recurrent neural networks (RNN) introduced a cyclic structure, which allows information exchange and sharing between hidden layers. Thus, it is capable of sequence data analysis and prediction. However, in the process of RNN training and testing, problems such as gradient disappearance, gradient explosion, lack of longterm memory capability etc. would occur [25]. A solution was proposed by Hochreiter et al. [26] and named as long short term memory (LSTM) structure which adds information flow path processing and analysis of sequence data better. The core of the LSTM structure is a unit with information storage function. Valuable information can be stored for a long time through the information flow. The forget gate, input gate and output gate in the LSTM (Fig. 5) have the functions of filtering the information on the flow and assigning weights to the new and old information, to selectively increase and remove the information in the cell. In simple terms, the forget gate determines which information should be abandoned from previous unit cell states, the input gate determines which input information and candidate cell state should be updated, and the output gate outputs information based on the state of the unit cell state. The vector formulas for a LSTM can then be written as: [27], [28] where f , i, o, c represent forget gate, input gate, output gate and cell, W is the input weights, U is the recurrent weights, b is the bias weights. σ is the sigmoid function, which enables the value between 0 and 1. tanh is hyperbolic tangent function, which enables the value between −1 and 1.

B. FEATURE EXTRACTION
The temperature curves were obtained by extracting the temperature of pixels in defective and non-defective area orderly, as shown in Fig. 6. The deeper the defects, the faster the VOLUME 8, 2020  temperature dropped. At the end, the temperature of all pixels would gradually drop to room temperature. The temperature evaluation of cooling stage was made as one of the data sets.
In addition, the fitting data obtained by using TSR was made as another data sets to train and test LSTM-RNN. TSR is one of the classic methods for processing pixelbased temperature evolutions. The surface temperature in the non-defective area can be described as one-dimensional diffusion equation [29]: which has the solution of where T is the surface temperature, Q is the energy absorbed by the surface, e is the thermal effusivity, t is the time after the laser excitation started. However, heat flow will be impeded when encountering internal adiabatic boundaries, such as air or similar defects. Heat must flow around the defect, so that the condition of one-dimensional diffusion no longer applies. It can be described by two-dimensional diffusion equation, in which case the solution in the log domain is: which is a straight line with slope −0.5 in the logarithmic coordinate system. In actual inspection course, due to the uneven laser power distribution and the influence of the internal structure of the material, the logarithmic data will be slightly shifted. A function or set of orthogonal functions can be used to approximate the logarithmic time dependence of a pixel. In this work, we used the fitted logarithmic data to amplify temperature discriminations between defective and non-defective areas. The 5th polynomial was used to fit thermal data, which can be described as: where the low-order polynomial was applied as an effective low-pass filter to maintain the basic thermal response, and the high-order only replicates noise that appears in the later, low amplitude data. The thermal signal can be reconstructed by Fig. 7 shows the cooling curves of defects with different depths after TSR. Compared with the raw data, we find that TSR has a better separation of temperatures in the last stage.
In order to prepare the training data sets, every thermal sequence was cropped and merged into a 750 × 300 thermogram, including all defective and part of non-defective areas, as shown in Fig. 8. For round defective areas in the red square, temperature evaluation of the pixels in the circle with the center of the areas were extracted. Out of the red squares, all pixels were considered as ones in non-defective area, and their temperature evaluation were extracted as well.

IV. RESULT AND EVALUATION A. RESULTS BASED ON LSTM-RNN MODEL
The LSTM-RNN model is illustrated by Fig. 9, which had one hidden layer containing 128 nodes. It was set based on trial and error. The output layer possessed 6 nodes for determining non-defective areas and defects of five depths. Due to the greater temperature change at the beginning of the cooling stage, too much information would be lost if a long-time step was set. For this reason, the time step of LSTM-RNN model was set to 2, which means that in the input layer vector group {x 1 , x 2 , . . . , x n }, the length of x i was 2. The vector x t combined with the hidden layer state h t−1 at the previous time step to be a vector of length 130 was used as the input of LSTM. Softmax activation function was applied in fully connected layer which contained 6 nodes to determine nondefective areas and defects of five depths. Adam optimizer  was used to update the weights of network. Batch size was set to 32.
We used the raw data and the TSR data obtained from the first experiment to train the network respectively. To compare the effect of the LSTM-RNN model trained by raw data set with the TSR data set in determining the depth of the defects, the parameters were the same. For convenience of description, the former was named as M1, and the latter was named M2. The data obtained from the second experiment was used to test the network. M1 started overfitting after 300 epochs training, in which case, the training loss decreased while the verification began to increase. Thus, M1 was trained for 300 epochs. The final training loss declined to 0.05 then stabilized. M2 was trained for 250 epochs, and final training loss declined to 0.03 then stabilized. Fig. 10 and Fig. 11 show that M1 and M2 both had high accuracy in the determination of defects with different depths (NDA represents non-defective area, numbers of 050 to 150 indicating that the defects depth ranging from 0.5mm to 1.5mm). The normalized confusion matrix was used to evaluate the training performance of the LSTM-RNN model ( Fig. 10(b) and Fig. 11(b)). The diagonal of the matrix represents the recall rates of the LSTM-RNN model, that is, the proportion of positive examples that were correctly predicted as positive [30]. The recall rates of M1 were above 90%  for the defects in 0.5mm, 0.75mm, 1.0mm. A possible reason is that, the heat accumulated around the deep defects emitted thermal radiation, which is attenuated greatly. Therefore, the thermal radiation signal received by the infrared camera was close to that at surface, resulting in a low recall rate of VOLUME 8, 2020 the model. The recall rates of M2 were above 95% for the defects in each depth. Therefore, in determination of defect depth, the effect of M2 is better than M1. This is because the infrared camera is sensitive to the interference from environment noise. It may cause fluctuations in the received thermal signal. The data after TSR retains the tendency of raw data, yet the smoothness and continuity have been greatly improved. Also, the temperature changes of defects in different depth discriminated much conspicuously.

B. COMPARISON WITH TRADITIONAL METHODS
In order to verify that the performance of the LSTM-RNN model in determining depth of defects by processing thermal signal, a RNN model and a CNN model were also trained and tested by raw data and TSR data to compare.
RNN is the general class of neural network, which is the predecessor of LSTM [31]. The model had one hidden layer, containing 128 nodes. Time step was also set to 2. Softmax activation function were applied. Adam optimizer was used for weight updating. Batch size was set to 32. The model trained by raw data was named as R1, and the model trained by TSR data was named as R2. R1 was trained for 200 epochs and loss was declined to 0.03. R2 was trained for 150 epochs and loss was declined to 0.02. Results are shown in Fig. 12 and Fig. 13.
CNN is a powerful deep learning method specializing in various fields. The core ideas of CNN include three parts: local receptive fields, shared weights, spatial or temporal  subsampling [32]. High-level features are extracted from the input data through operations such as convolution operations,  pooling operations, and nonlinear activation function mapping [33]. A CNN model was applied for depth determination. Each convolutional layer contained 64 convolutional kernel, of which size was 3 × 1. A maxpooling layer was added every two convolutional layers to reduce the massive data calculation. Softmax activation function was used, and Adam optimizer was applied to update weight of network. Batch size was set to 32. The model trained by raw data was named as N1, and the model trained by TSR data was named as N2. N1 was trained for 100 epochs and loss was declined to 0.03. N2 was trained for 80 epochs and loss was declined to 0.05. Results are shown in Fig.14 and Fig. 15.
Results show that, the recall rates of R1 are above 80% for the defects in 0.25mm to 1.5mm, and those of R2 are nearly 90%. However, the recall rates of N1 are below 80% for the defects in 0.25mm to 1.5mm. The recall rates of N2 are 84% for the defects in 1.0mm and are around 80% for the defects in 0.25mm, 0.75mm, 1.25mm, 1.5mm. Therefore, we find LSTM-RNN>RNN>CNN in terms of the recall rates. Because of the specialty in processing time related sequence data, RNN testing results is close to LSTM-RNN. CNN performs well in extracting local features, but compared with the other networks, it has insufficiency in processing time related sequence data.

V. CONCLUSION
In this work, LIT combined with deep learning algorithm has been utilized to achieve the depth determination of the debonding defects inside CFRP laminate. The raw data collected by infrared camera and the data processed by TSR have been used as input to train and test LSTM-RNN model. Using the same method, the model trained by TSR data set outperforms the model trained by raw data set, indicating that environment noises in the raw data can be effectively reduced by TSR method and the continuity of the temperature evolution sequence of defects at different depths were enhanced. At the cooling stage, TSR data showed the greater discrimination of temperature changes of defects at different depths, which is important for the automatic determination of depth. The performance of LSTM-RNN model has been evaluated and compared with a RNN model and a CNN model. In terms of the recall rates, the LSTM-RNN performs better than the other two methods. In other words, for the thermal sequence signals, the memory mechanism of LSTM-RNN enables it to learn more signal characteristic information than RNN and CNN. We expect this work will advance the field of NDT using AI algorithms to improve the efficiency and accuracy of defect identification and locating.
QIUHAN LIU is currently pursuing the degree in control engineering with Air Force Engineering University, Xi'an, China. His current research interests include application of terahertz and infrared in non-destructive testing of carbon/glass fiber composites and defect classification using deep learning algorithm. He is currently a Lecturer with the Aviation Safety and Airworthiness Laboratory, Air Force Engineering University. His research direction is aviation safety management and system engineering.

RUICONG
HONGBIN ZHOU received the B.S. degree in flight vehicle propulsion engineering and the M.S. degree in control science and engineering from Air Force Engineering University, Xi'an, China, in 2017 and 2019, respectively, where he is currently pursuing the Ph.D. degree in control science and engineering.
Since 2017, he has been a Research Assistant with the Aviation Safety and Airworthiness Laboratory, Air Force Engineering University. His research interests include the deep learning and terahertz technology.
Mr. Zhou's awards and honors include the two articles and two inventions from 2017 to 2019.
BOYAN ZHAO received the B.B.A. degree in management engineering from Air Force Engineering University, Xi'an, China, in 2019, where he is currently pursuing the master's degree in control science and engineering. He has been a member of the Aviation Safety and Airworthiness Laboratory, Air Force Engineering University, since 2019. His research direction is deep learning and NDT technology. VOLUME 8, 2020