Terahertz super-resolution algorithm for fusible networks based on edge feature convolution

Much research has been conducted to improve the defect-detection rate and detection accuracy of the imaging technology used in terahertz nondestructive testing. Due to the power limit of light sources and noise interference in terahertz equipment, images have low resolution and fuzzy defect edges. Hence, improving the resolution is crucial for detecting defects. We designed an edge detection network structure based on a traditional deep neural network. Besides, we devised a node-fusing strategy to train the network. It demonstrates significant improvement of the resolution of the terahertz defect contour. A quartz fiber composites with embedded defects was tested with our network. The results showed that the proposed super-resolution reconstruction algorithm improves resolution, particularly on the edges of defect contours.


I. INTRODUCTION
One goal of a single-image super-resolution (SR) algorithm is to find the mapping relationship between highresolution (HR) and low-resolution (LR) images, and improves many image-processing and analysis methods.Mapping relationship is nonlinear and nonadaptive, i.e., a low-fraction image can correspond to numerous HR images, most of them unexpected.The reconstructed HR image should meet visual expectations and be as close to the real image as possible [1][2].
Terahertz time-domain spectroscopy technology is the major method in Terahertz nondestructive testing (NDT).The time-domain signal of each point of the sample represents the spectral information.The terahertz frequency spectrum can be obtained by a fast Fourier transform, and a terahertz image can be obtained by a correlation imaging algorithm.The defects on the surface and in the sample can then be identified from the image according to the characteristic parameters of the terahertz spectrum difference between the defect and sample.However, image quality depends on the obtained time-domain signal.Background noise and hardware vibration will affect a terahertz signal and degrade image quality in terms of low image contrast, a blurred defect outline, and difficulty finding small-size defects.
Much research SR algorithms and improved network structures based on deep learning have been reported.Super-resolution convolutional neural network (SRCNN) is a typical fully connected convolutional neural network (CNN) that is widely used in single-image SR algorithms.VDCNN adds longitudinal network depth on the basis of SRCNN to improve the complex coefficient of the mapping function and provide a better SR effect.Residual channel attention network (RCAN) combines an attention mechanism and a residual network with a CNN structure to enhance learning ability.Enhanced deep super-resolution (EDSR) residual network for single image removes some modules from the traditional residual network structure, thereby improving the application effect in problems of low visual scale, such as SR.However, only a few methods have studied the scanning image of terahertz NDT.
Because terahertz images are usually imaged by spectral characteristics and optical characteristic parameters, they differ from most imaging methods.In addition, bad point signal data may cause gradient explosion or disappearance during network training, resulting in serious damage to network weight parameters.The core problem of terahertz NDT image SR is to find the relationship between LR and HR images.At the same time, the sensing field is placed as close as possible to the center of the images.

A. DESCRIPTION OF TERAHERTZ SINGLE-IMAGE SUPER-RESOLUTION PROBLEM
Let A and B represent HR and LR images, respectively.The conventional SR reconstruction problem can be expressed as  =  +  (1) where H is the synthesis operator from an LR to an HR image, and N is additional noise.The purpose of SR reconstruction is to obtain the mapping of HR images from LR images.Because we want to find the mapping of SR terahertz images, we set the weight parameter matrix of the CNN as  = ( − ) −1 (2) We can then obtain  = ( − ) −1 +  (3) where X and Y are the input and output matrices of the network, respectively, and B is the amount of deviation.Because the CNN has an activation function, the input and output of the network have a nonlinear relationship, which conforms to the characteristic of inadequate mapping from LR to HR images.Then, we can calculate the weight matrix by constructing the appropriate training dataset.

B. DATASET CONSTRUCTION
Because the loss function of CNN is related to the difference between HR and LR images, it is necessary to create an LR image sample based on an HR image.For better experimental results, we carried out terahertz scanning imaging through several pieces of embedded defect quartz fiber composite material, and we also carried out gray imaging by considering the time-domain amplitudes of different moments, for a total of 150 images.In order to improve the quality of the data set, we also increased the number of images used in the experiment to 1241 by Image enhancement.The HR image size is then zoomed at scales of 2, 3 and 4 by cubic interpolation downsampling, and traditional cubic interpolation is used to restore the reduced LR image size to obtain the LR image.Groups of LR images with zoom scales of 2, 3 and 4 and their corresponding HR images are used as datasets.In addition, the defect contour is extracted from the terahertz defect image by edge detection, and the binary defect contour image is obtained by the binary image projection as the feature convolution extraction image set.

III. EFCFN Network
The structure of the EFCFN network is shown in Fig. 1.It consists of edge detection convolution neural network (EDCNN), downsampling convolution neural network (DSCNN), and circuit breaker convolution neural network (CBCNN).The network works as follows: 1) EDCNN extracts edge feature of defects and outputs the binary images of edge detection.2) The binary images are resized to a fixed size as we required that would be used as convolution kernels by the DSCNN.3) CBCNN is trained by data set with circuit breaker strategy.4) LR images are reconstructed to HR images by trained CBCNN.
Finally, the mapping weight matrix from the LR to the HR image is obtained by CBCNN, whose node fusing mechanism can protect the gradient from smooth descent during training.
EFCFN differs from the traditional SR algorithm based on CNN and derivative algorithms such as SRCNN [19], FSRCNN [22], VDSR [23], and DnCNN [29].EFCFN has a feature module that is aimed at the terahertz defect image.Most SR algorithms deal with conventional images, particularly those based on deep learning methods.The EFCFN network architecture differs from that of other networks.This is partly due to the EDCNN, DSCNN and CBCNN modules, but the biggest difference between EFCFN and other networks lies in its convolutional accounting subsets with edge feature extraction and training strategy.The features of EFCFN that differentiate it from other networks include the following: • It focuses on the characteristics of defect images.To extract the defect features well, an edge detection module extracts defect contour features, which are resized by the downsampling module and used as the convolution kernel in CBCNN.This method can also be applied to image SR.

•
It has three cascaded modules with different functions, which can carry out phased output according to different requirements and which have a specific generalization ability.• It uses a new type of training that enables the network's modules to achieve their respective functions.In addition, a node circuit breaker mechanism makes the network weight training more efficient and accurate.
In the EFCFN, the specific features of the terahertz defect image can be accurately extracted.Experimental results confirm its advanced SR performance in terahertz defect images.

A. NETWORK STRUCTURE
As shown in Fig. 1, the input of the EFCFN consists of two parts.The contour image of the terahertz binary graph is fed into the EDCNN module as a convolution training set (① in Fig. 1).Its output ② is the input of DSCNN, whose output is used as the convolution kernel of CBCNN after downsampling.The terahertz imaging dataset ③ is the input of CBCNN, which is used to train the mapping weights between HR and LR images.The edge operator is the core of the edge detection network.We will compare and analyze the detection results of several edge detection operators.The operator is selected by combining the terahertz imaging results of the sample.
Considering that the data have multi-scale images, we chose to use the network implementation method of HOLA-nested networks [27].As shown in Fig. 2, four branches are generated through side branch and fuse branch functions and then as the output of the network through the activation function.Each output image has the same size of the input image.The side branch and fuse branch functions are respectively ̂ = (∑ where  ̂ () is the output of each branch, Y is the output of the CNN,  ̂ () is the side branch output after the activation function, and  is the weight of each branch.We set all network parameters to  and side branch parameters to  = ( (1) , … ,  () ); then, the loss function of the side branch can be defined as The binary edge contour image of terahertz defect imaging is extracted as the output of the network module.

2) DSCNN
The downsampling module consists of two types of network layers.The first is the same convolution layer as EDCNN, which uses 32 convolution kernels of size 5 × 5 and conducts batch standardization and ReLU.It can be described as  _ (;  _ ,  _ ) = (   *   −1 () +    , 0) where  = 1, 2, … ,  − 1 is the index of the convolution layer in DSCNN,  _ is the k-th output, Y is the initial input  _0 =  of the DSCNN,  _ is the weight matrix of the convolution layer, and  _ is the offset.The first  − 1 layers of the DSCNN are the same as those of a typical CNN, and the last layer () performs downsampling to determine the size of the convolution kernel, which is also the characteristic convolution kernel in the CBCNN, where    () is the final output, and ∇ is the downsampling operator.

3) CBCNN
CBCNN has a network structure similar to that of a typical CNN.To avoid repetition, we will not describe it in detail.The main feature of this module is that it combines the weight circuit breaker strategy, which will be described below.(12) where Θ  is the trainable parameter in CBCNN, and   is the fusing function.Since we use the value of edge feature to initialize the weight instead of random initialization.The gradient may be extremely large after multiplying by the unknown weight.In addition, due to the depth of the network model, the gradient may disappear at the same time.The   could set a threshold range.When the weight of the node exceeds the threshold range, the back propagation of the node will be stopped automatically.In this way, the adverse effects of gradient explosion and gradient disappearance can be alleviated.The setting of the threshold value needs to be continuously adjusted according to the actual experimental effect.How to set the threshold is also one of the topics we are studying.

B. NETWORK TRAINING STRATEGY
The SR reconstruction of terahertz defect images can be achieved through the three modules with edge detection, downsampling and SR reconstruction, and the above training strategy, respectively.If we train neural networks through traditional methods, the above problems will appear and affect the SR effect.We achieved stable SR by improving the network structure and training strategy.

IV. EXPERIMENT AND RESULTS
We describe the equipment used in our experiment and the samples, and then provide the experimental results, including quantitative evaluation indicators and visual effects, to verify the effectiveness of EFCFN.

A. EQUIPMENT
As shown in Fig. 3(a), the terahertz scanning device uses a self-assembled terahertz time-domain spectrometer with a 6-DOF mechanical arm structure.Fig. 3 is transmitted through the optical fiber to a photoconductive antenna to stimulate the THz wave, and the optical path of the fiber-coupled THz time-domain spectral system is completed by combining the optical delay line based on the voice coil motor platform.The maximum output power of the device is about 110 mW, the central wavelength is 1550 nm, and the pulse width is less than 90 FS, which meets the requirements of terahertz NDT.

B. EXPERIMENTAL SAMPLE DESIGN
As shown in Fig. 4, the experimental sample design uses fiber composite laminate, and the red dotted line on the image is marked as the outline of embedded PTFE defects.The sample size is 120 mm × 120 mm and is composed of four laminated layers with a thickness of 0.4 mm.Between each layer are PTFE sheets with a diameter of 20 mm and a thickness of 0.02 mm.

C. EXPERIMENTAL RESULTS
The depth parameters of EDCNN, DSCNN, and CBCNN were 5, 3 and 15, respectively.The degradation model employed bicubic interpolation.The HR terahertz image was downsampled and amplified by the  function in MATLAB.The SR algorithms that were compared were bicubic, FSRCNN [22], A+ [16], VDSR [23] and ARCNN-VDSR [28].To ensure fair comparisons, we set the network depth of VDSR and ARCNN-VDSR to be the same as ours.We compared performance, robustness and efficiency.The test dataset was obtained by terahertz scanning imaging of 20 objects.We evaluated the performance by an objective quantitative evaluation index and a subjective visual effect evaluation.Because this method is mainly aimed at the application of terahertz technology to NDT, the quantitative evaluation metrics include peak signal-to-noise ratio (PSNR), structural similarity (SSIM), information fidelity (IFC), minimum defect detection resolution (MDR), robustness and reconstruction time.Due to space limitations, only good imaging results are presented.To evaluate robustness, we used the images generated by terahertz signals with different noise influence factors for SR and compared the PSNRs of the algorithms.Efficiency was compared through the running time of SR reconstruction.
Fig. 5 shows the train loss curve in our experiment.Convergence occurs after 50 iterations.The value of loss tends to be 0.2.Generally, the loss is at a small value and no longer significantly smaller, so we think the network has been trained enough.

1) PERFORMANCE INDEX EVALUATION
As shown in Table 1, in terms of network performance indicators, the EFCFN achieves the highest MDR and the second-highest PSNR, SSIM and IFC, while the ARCNN-VDSR has the second highest.VDSR and FSRCNN are similar in performance, and A+ has slightly better performance than bicubic.
The noise contained in the image significantly degraded the performance of A+, which is sensitive to noise.VDSR and ARCNN-VDSR, two of the most advanced SR algorithms, both showed a significant gain in natural image SR reconstruction, surpassing EFCFN on some evaluation indices.Of course, the SR performance of EFCFN to a degree benefit from the similarity between the final CBCNN module structure and the VDSR network.However, because VDSR and ARCNN-VDSR lack pertinence for defects on terahertz images, they do not perform as well as EFCFN in the MDR index.In the objective evaluation indices, although EFCFN is slightly inferior to VDSR and ARCNN-VDSR on PSNR, SSIM and IFC, it ranked highest in MDR in the field of NDT, achieving superior SR performance.
Fig. 6 shows six different edge detection operators to compare and verify results.They are Roberts-operator, Prewitt-operator, Sobel-operator, LOG-operator, Cannyoperator and wavelet edge detection.The results show that the processing results of Roberts, Prewitt and Sobel operators are close to each other, and the outline of defects can be roughly identified, but it cannot be connected into a complete circle.The LOG-operator calculation diagram can identify the rough outline of the defect, while the Cannyoperator calculation diagram cannot identify the defect.The "wavelet edge detection" algorithm shows the defective circular contour very well.Therefore, we adopt wavelet edge detection operator as EDCNN feature.
The visual evaluation in Fig. 7 shows that EFCFN achieves a significant improvement in the quality of detail texture and defect contour, with fewer artifacts and a clear structure.VDSR and ARCNN-VDSR are also visually excellent, while FSRCNN and bicubic are less effective.All in all, EFCFN achieves a strong gain in image quality in the field of terahertz defect imaging, comparable to the most advanced SR imaging algorithms.

2) NETWORK ROBUSTNESS EVALUATION
We tested the robustness of the networks by reprocessing the terahertz signal by adding different noise factors, with a larger noise factor indicating greater interference.Fig. 8 shows the PSNR distribution of network algorithms under different noise factors.It can be seen that EFCFN resists noise well and achieves an obvious image gain effect due to its node-fusing training mechanism.This means that EFCFN is suitable for SR reconstruction in cases of high background noise.

3) NETWORK EFFICIENCY EVALUATION
Computation time is another key evaluation metric.We compared the computation times of network methods under the same scaling factor, including training to convergence and SR reconstruction.
As shown in Fig. 9, A+ is the most efficient method.EFCFN consumes more computation time than VDSR and ARCNN-VDSR because its EDCNN and DSCNN modules account for additional calculation overhead.As a result, the training time of EFCFN is much longer than that of other methods, but its SR reconstruction time is similar, and it is more stable.For example, EFCFN's computing time is roughly the same as that of FSRCNN, but it has better stability.Complex network structure brings huge parameter calculation.EDCNN module needs to pre-extract the features of the data.Since the operation process of EDCNN and CBCNN can't be calculated at the same time, low efficiency of the network limitation is exposed.Overall, EFCFN achieves state-of-the-art SR reconstruction of terahertz defect images in an acceptable computational time.

V. CONCLUSION
We propose a learning-based super-resolution algorithm for terahertz images.Unlike other learning-based methods, our EFCFN algorithm combines terahertz image features with the network convolution kernel.In addition, we designed a training mechanism of a fusing network, which can avoid the problems of gradient disappearance and explosion.Experimental verification was carried out by terahertz scanning imaging of quartz fiber composite samples with embedded defects, which showed that our proposed EFCFN achieved the highest SR results in defect detection.EFCFN can further improve the terahertz resolution limit when a scanning step is allowed.In addition, when compared with other SR algorithms on many evaluation metrics, EFCFN gained the highest quality of the defect contour and the sharpest detail when it achieved acceptable computational efficiency.However, complex network structure brings huge parameter calculation.Usually, EDCNN needs to be retrained in different tasks.The optimizing for EFCFN including the number of layers, the number of convolution kernels, and kernel size is necessary.Our next work will focus on the parameter settings of EFCFN to reduce its complexity and improve its performance.However, the number of samples we have is extremely limited.It's not good for verifying the network model.More experimental designs are needed in our next work.
At present, terahertz nondestructive testing technology is used as an important supplementary technique in defect detection of composite materials and nonpolar materials.Imaging resolution is closely related to scanning step.On the one hand, high resolution images cost a lot of time.On the other hand, most imaging methods are not effective for defect detection.Therefore, the network we proposed could improve image resolution by algorithm instead of reducing scan step.So that we could greatly shorten the required time of terahertz scanning sample, improve the value of industrial applications.Depressingly, we do not have any commercial application for this network for the time being.
Based on the above, we propose an edge feature convolution fusing network (EFCFN) for single-image SR reconstruction of terahertz NDT images.The four main contributions of our research are: • We design EFCFN, which introduces characteristic convolution on the basis of a traditional CNN.The improved network could focus on the defect contour and effectively on sharpening the edge.• We use a special optimization training strategy in network training.The weight node fusion is to improve the training speed and reduce the influence of bad data.• An independent convolution module brings a targeted optimization on terahertz NDT.Edge feature replaces the traditional network convolution kernel, and solving the instability caused by random initialization weights.• Experimental results show that EFCFN achieves excellent SR performance in single-image SR reconstruction on the design sample.The rest of this paper is organized as follows.Section Ⅱ reviews relevant work.Section Ⅲ introduces the network structure and training mechanism of EFCFN.Section Ⅳ discusses our experiments and their results.Section Ⅴ summarizes the paper and suggests directions for future research.

FIGURE 2 .
FIGURE 2. Schematic diagram of branch fusion algorithm.
1) EDCNNConvolution kernel represents the feature mapping of image usually.The parameters of convolution kernel are generally randomly initialized.Therefore, the extraction of image features is generalized instead of pertinence.An independent convolution module could extract the edge features of defects first.Then the extracted feature map is used for the initialization of convolution kernel through down sampling module.Compared with random initialization, it has targeted optimization effect on terahertz image defect detection.
This article has been accepted for publication in IEEE Access.This is the author's version which has not been fully edited and content may change prior to final publication.Citation information: DOI 10.1109/ACCESS.2022.3184029This work is licensed under a Creative Commons Attribution 4.0 License.For more information, see https://creativecommons.org/licenses/by/4.0/VOLUME XX, 2017

,
6) where   is the weight of each side branch loss function, which can be adjusted according to the training log, and ℓ () (, () ) is the loss function of each side branch.The loss function adopts the class-balanced cross-entropy loss function and is expressed as ℓ  () (,  () ) = − ∑ (  = 1|; ,  () ) | + | is the number of edge pixels and | − | is the number of non-edge pixels.The training objective functions ℒ  (, ) and ℒ  (, , ℎ) are:(, , ℎ) * =  (ℒ  ( + ℒ  (, , ℎ))) (b) shows the structure of the laser optical path.A femtosecond laser This article has been accepted for publication in IEEE Access.This is the author's version which has not been fully edited and content may change prior to final publication.Citation information: DOI 10.1109/ACCESS.2022.3184029This work is licensed under a Creative Commons Attribution 4.0 License.For more information, see https://creativecommons.org/licenses/by/4.0/

FIGURE 4 .
FIGURE 4. Experimental sample and design.

FIGURE 5 .
FIGURE 5.The loss curve in experiment.
(a)Roberts-operator (b)Prewitt-operator (c)Sobel-operator (d)LOG -operator This article has been accepted for publication in IEEE Access.This is the author's version which has not been fully edited and content may change prior to final publication.Citation information: DOI 10.1109/ACCESS.2022.3184029This work is licensed under a Creative Commons Attribution 4.0 License.For more information, see https://creativecommons.org/licenses/by/4.0/VOLUME XX, 2017 (e)Canny-operator (f)wavelet edge detection

ACKNOWLEDGMENT
This article has been accepted for publication in IEEE Access.This is the author's version which has not been fully edited and content may change prior to final publication.Citation information: DOI 10.1109/ACCESS.2022.3184029This work is licensed under a Creative Commons Attribution 4.0 License.For more information, see https://creativecommons.org/licenses/by/4.0/