A Low- and High-Resolution Terahertz Image Pair Construction Method With Gradient Fusion for Learning-Based Super-Resolution

Quality and feature quantity of low- and high-resolution image pairs for training are directly related to performance of learning-based super-resolution methods. In order to accurately increase learning features to the low- and high-resolution image pairs, we proposed a gradient degradation model of terahertz(THz) images to describe the process of a high-resolution THz image’s gradient map is how degraded to corresponding low-resolution THz image’s gradient map. And with the proposed model, we presented a low- and high-resolution THz image pair construction method with gradient fusion for learning-based super-resolution. As gradient maps are fused to training pairs, performance of learning-based super-resolution methods for THz images could be improved. In addition, we applied the low- and high-resolution THz image pair construction method with gradient fusion to very deep super-resolution(VDSR) method and zero shot super-resolution(ZSSR) method, named VDSR method with gradient fusion and ZSSR method with gradient fusion, respectively. To evaluate the performance of these two improved methods, comparison experiments with passive millimeter-wave images and our THz lab data are presented. The experimental results show that the performance of the VDSR method with gradient fusion and the ZSSR method with gradient fusion have significant improvements in peak signal-to-noise ratio and structural similarity index measure than the VDSR method and the ZSSR method, respectively. It indicates that performance of learning-based super-resolution methods for THz images could be improved by applying the proposed low- and high-resolution THz image pair construction method with gradient fusion.

imaging system and the intrinsic long wavelength of THz wave. Therefore, it is significant important to improve the resolution of THz images.
There are two main approaches to improve the spatial resolution of THz images, one is to enhance the equipment for THz imaging system [13], [14], [15], and the other is to adopt a super-resolution method [16], [17], [18], [19], [20]. Obviously, the first approach is complicated and expensive. The other approach is a more effective and flexible solution to obtain high-resolution THz images only using existing imaging system. Therefore, many super-resolution methods for THz image have been proposed to improve reconstruction performance. One of the most frequently used methods for super-resolution reconstruction is the Lucy-Richardson method, which is a type of deconvolution superresolution method. Nevertheless, the object's edge in the super-resolution reconstructed image is prone to be blurred in the process of increasing image size with the Lucy-Richardson method. In addition, due to the degradation model of THz imaging system is different from that of the optical imaging system, and learning-based super-resolution methods have achieved state-of-the-art super-resolution reconstruction quality for optical images, a few learning-based super-resolution methods for THz images have been proposed to obtain better performance in both objective index and visual quality [19], [20], [21].
The main goal of a learning-based super-resolution methods for THz images is to obtain accurate mapping between low-and high-resolution THz images. For this purpose, degradation model of THz images is constructed to express low-resolution THz image is how generated from a high-resolution THz image, and a deeper and more complex neural networks are proposed in [19]. According to the proposed degradation model for the frequency-modulated continuous-wave(FMCW) THz imaging system [19], the high-resolution THz image is first blurred by a point spread function(PSF), then down-sampled by a down-sampler, and finally added by a Gaussian white noise. The processing result is the low-resolution THz image. Meanwhile, compared with neural networks such as the super-resolution convolutional neural network (SRCNN) [22] and the superresolution using very deep convolutional networks(VDSR) [23], the neural network proposed in [19] is deeper with stronger feature extraction ability and better super-resolution reconstruction performance. However, the degradation model of THz images proposed in [19] is just suitable for the focal plane THz imaging system. Then a 3D degradation model for THz imaging system is proposed in [20]. According to the 3D degradation model, the high-resolution THz image is blurred by a 3D PSF, then down-sampled by a down-sampler, and finally added by a Gaussian white noise. As it is hard to accurately obtain the 3D PSF distribution experimentally, the 3D PSF of the typical THz imaging system is approximated by a TEM00 mode Gaussian bean. And an adjustable deep residual convolutional neural network(Res-CNN) was proposed to solve the accommodative super-resolution problem for THz images [20]. Furthermore, the reliability of the PSF mathematical model shown in [20] is proved with an experiment in [24]. On account of the features of THz images are implicit, a inception module is designed to enhance the feature extraction ability of their proposed network [24].
According to the analysis above, the super-resolution reconstruction performance of existing learning-based superresolution methods for THz images is affected by two main aspects. One is the type and structure of neural networks. The other is the accuracy of degradation model of THz images.
Nevertheless, above degradation model is just suitable for the THz imaging system, where the high-resolution THz image is blurred by PSF, but not suitable for linear scanning THz imaging system [25], [26] or focal plane array THz imaging system [27].
To address the insufficient generality of degradation model of THz images in existing learning-based super-resolution methods for THz images, we considered enriching learning features included in training datasets to improve superresolution performance. As edge feature includes a wealth of high-frequency information which benefit to reconstruct high-resolution images, we added edge features of training images to the training datasets. In the existing learningbased super-resolution methods for THz images, low-and high-resolution image pairs are included in the training datasets. And low-resolution images were obtained from high-resolution images with a degradation model of THz images. Therefore, in order to add edge features to the training pairs, we focused on analyzing the degradation process when edge features pass through THz imaging system and proposed a gradient degradation model of THz images to describe the degradation process of high-resolution THz image's gradient maps. In general, the gradient maps are the most important edge features in super-resolution methods.
Benefit from the proposed gradient degradation model of THz images, we also presented a low-and high-resolution THz image pair construction method with gradient fusion for learning-based super-resolution. In the proposed construction method, a THz image is fused with its horizontal and vertical gradient maps to constitute a three-channel image which is the high-resolution image in training datasets. Meanwhile, according to the proposed gradient degradation model of THz images, low-resolution THz image's vertical and horizontal gradient maps are obtained from corresponding high-resolution THz image's vertical and horizontal gradient maps respectively. And low-resolution THz image is obtained according to general degradation model of THz images. Then the low-resolution THz image and its vertical and horizontal gradient maps are fused to form the low-resolution image in training datasets.
With the proposed low-and high-resolution THz image pair construction method with gradient fusion, we improved the VDSR method which is one of multi-image superresolution method, and proposed the VDSR method with gradient fusion. Meanwhile, zero shot super-resolution(ZSSR) method, which is a popular single-image super-resolution method for optical images, is also improved in our work and the ZSSR method with gradient fusion is proposed.
In summary, the main contributions of our study are as follows.
• A gradient degradation model of THz images is proposed. With the proposed gradient degradation model of THz images, a low-resolution THz image's gradient map could be accurately deduced from corresponding high-resolution THz image's gradient map. VOLUME 10, 2022 • A low-and high-resolution THz image pair construction method with gradient fusion is presented for learning-based super-resolution. With the proposed training pairs' construction method, vertical and horizontal gradient maps are added into training images, so that the performance of learningbased super-resolution methods for THz images could be effectively improved.
• VDSR method and ZSSR method are improved by applying the low-and high-resolution THz image pair construction method with gradient fusion, and the VDSR method with gradient fusion and the ZSSR method with gradient fusion are proposed. These two improved super-resolution methods for THz images outperform the VDSR method and the ZSSR method for THz images, respectively.
The remainder of this paper is organized as follows. Section II presents the gradient degradation model of THz images to describe degradation process of high-resolution THz image's gradient maps. In section III, the low-and high-resolution image pair construction method with gradient fusion for learning-based super-resolution is introduced in detail. Then the VDSR method with gradient fusion and the ZSSR method with gradient fusion are presented in section IV. In section V, the super-resolution reconstruction performance of VDSR method with gradient fusion is validated by comparing the performance of VDSR method with gradient fusion, VDSR method, and Lucy-Richardson method. As well, the super-resolution reconstruction performance of ZSSR method with gradient fusion is validated by comparing the performance of ZSSR method with gradient fusion, ZSSR method, and Lucy-Richardson method. Section VI presents the conclusion of this study and our future work.

II. GRADIENT DEGRADATION MODEL OF THZ IMAGES
The goal of learning-based super-resolution method for THz images is to restore the high-resolution image from a lowresolution image. Then the neural network in learningbased super-resolution method should provide an end-to-end mapping between low-and high-resolution images. Obviously, in order to obtain the end-to-end mapping, low-and high-resolution image pairs should be firstly constructed for neural network's training. Thus, the general degradation model of THz images has been constructed [19] to obtain a low-resolution image from a high-resolution image, which could be expressed as follows.
where y is the low-resolution THz image, x is the highresolution THz image, k represents the blur kernel, ⊗ is the convolution operation, D s is the down-sampler with scale factor s, and n is the additive noise with noise level α.
Meanwhile, the mapping performance of a neural network could be improved by adding features in training pairs, i.e., the edge information. Hence we constructed a gradient degradation model of THz images to obtain low-resolution image's gradient map from corresponding high-resolution image's gradient map for neural network's training.
Generality, the blurred high-resolution THz image with blur kernel k is down-sampled by bicubic down-sampler with scale factor s [19], [20]. So equation (1) could be written as According to images' derivative theorem of convolution, the gradient of low-resolution image y in (2) is deduced, which is shown as follows.
where ∇y is the gradient of low-resolution image y, ∇x is the gradient of high-resolution image x, k represents the blur kernel, ⊗ is the convolution operation, D s is the downsampler, and ∇n is the gradient of the additive noise n. According to (3), a low-resolution THz image's gradient could be accurately calculated with corresponding high-resolution THz image's gradient, whose degradation process is expressed by (2). Meanwhile, we can obtain from (3) that low-resolution THz image's gradient is not equal to the degradation result of corresponding high-resolution THz image's gradient. In the following section, with the proposed gradient degradation model of THz images, a low-and high-resolution image pair construction method with gradient fusion is introduced in detail for the purpose of adding edge feature to neural network's training datasets.

III. LOW-AND HIGH-RESOLUTION THZ IMAGE PAIR CONSTRUCTION METHOD WITH GRADIENT FUSION
In learning-based super-resolution methods, the first step is training datasets' construction, which is directly related to the mapping performance of neural networks. In existing construction process of training datasets [19], images in benchmark image datasets are converted into the YCbCr color space and the Y channel is selected as the high-resolution image, namely the images in benchmark image datasets are converted into grayscale images, which are the high-resolution image. Then the high-resolution images are degraded by using (2) to obtain low-resolution images, which are the input data of the neural network in learning-based super-resolution methods.
In the proposed low-and high-resolution image pairs' construction method with gradient fusion, the high-resolution image is firstly converted to grayscale image, then the grayscale image is filtered by Sobel filter to obtain x-direction and y-direction gradient maps of the grayscale image. The Sobel filter is a popular method to calculate the partial derivatives of a image, thus we used a pair of 3 × 3 Sobel filter, one estimating the gradient in x-direction to obtain the horizontal gradient map, and the other in y-direction to obtain the vertical gradient map. Due to rich edge features in the horizontal and vertical gradient maps, we combine them with the grayscale image to form a three-channel high-resolution image. The combine process is as follows: the grayscale image overlies on its horizontal gradient map. Then they overlie on the grayscale image's vertical gradient map to form the three-channel high-resolution image. To get low-and high-resolution image pairs for training neural network, the grayscale image in the three-channel high-resolution image is processed according to (2), and the other two gradient maps in the three-channel high-resolution image are degraded according to (3). These three degraded images are fused to a three-channel low-resolution image, which is the input data of neural network in learning-based super-resolution methods. Above low-and high-resolution image pairs' construction process could be illuminated in Fig 1. Owing to the rich edge features in low-and high-resolution image pairs constructed in Fig.1, we applied the low-and high-resolution image pairs' construction method with gradient fusion to VDSR method which is one of multi-image super-resolution method. Meanwhile, we applied the lowand high-resolution image pairs' construction method with gradient fusion to ZSSR method, which is a popular singleimage super-resolution method for optical images. The detail of these two improved super-resolution methods is presented in the following section.

IV. VDSR METHOD WITH GRADIENT FUSION AND ZSSR METHOD WITH GRADIENT FUSION
Benefit from the rich edge features in the low-and highresolution image pairs constructed by the proposed method above, we modified the VDSR method and the ZSSR method, which are typical supervised SR method and typical unsupervised super-resolution method, respectively. And we proposed VDSR method with gradient fusion and ZSSR method with gradient fusion for THz images.
The VDSR method with gradient fusion for THz images includes three steps. The first step is construction of training datasets. In this step, all images in datasets of 291 are converted to gray images. Then, the construction process of lowand high-resolution image pairs are according to Fig.1.
The second step is training the designed residual convolutional neural network(ResCNN) using low-and high-resolution image pairs constructed in the first step. In training step, the input low-resolution image is interpolated to the output size as done in the previous VDSR method [23]. Then, the interpolated image is mapped to an estimation of the high-resolution image. In our method, a fully convolutional ResCNN is adopted, with 20 layers, each of which has 64 channels. Besides, ReLU activation is used on the first 19 layers, while no activation was applied on the last layer. Furthermore, L1 loss is used with the Adam optimizer. Moreover, we trained all experiments over 80 epochs.
The third step refers to reconstructing high-resolution THz image with the ResCNN trained in step two. As shown in Fig. 2, x-direction and y-direction gradient maps of the super-resolution reconstructed THz image are calculated, and they are fused with the super-resolution reconstructed THz image to form a three-channel image. After that, the threechannel image is normalized. Then, the high-resolution THz image is reconstructed with the ResCNN trained in step two. On the whole, steps of the VDSR method with gradient fusion is almost the same with that of the VDSR method [23], except for the dimension of training data and superresolution reconstructed image. The dimension of training data and super-resolution reconstructed image in the VDSR method is one, but the dimension of training data and superresolution reconstructed image in the VDSR method with gradient fusion is three. Benefit from more edge information contained in training data in the VDSR method with gradient fusion than the VDSR method, the mapping performance of ReCNN could be improved for the VDSR method with gradient fusion. Therefore, the super-resolution reconstruction performance of the VDSR method with gradient fusion will be better than that of the VDSR method theoretically.
The ZSSR method with gradient fusion for THz images is one of single image super-resolution methods. Thus, the most difference between the ZSSR method with gradient fusion and the VDSR method with gradient fusion for THz images is the source of training datasets. In the ZSSR method with gradient fusion, the source of training datasets is the superresolution reconstructed THz image. Nevertheless, the source of training datasets in the VDSR method with gradient fusion is the benchmark image datasets.
In the first step of the ZSSR method with gradient fusion for THz images, which is illustrated in Fig. 3, the threechannel low-and high-resolution training images are firstly constructed according Fig. 1. After that, the three-channel high-resolution THz image is randomly cropped, and the low-resolution THz image is cropped in terms of the cropping region in high-resolution image. Then the cropped low-and high-resolution image pairs are the training data for the ZSSR method with gradient fusion.
Meanwhile, to enrich low-and high-resolution image pairs for training, each low-and high-resolution image pairs is transformed using 4 rotations (0 • , 90 • , 180 • , and 270 • ) as well as their mirror reflections in the vertical and horizontal directions. In this case, ×8 more image-specific training pairs are added.
The second step of the ZSSR method with gradient fusion for THz images is training a designed ResCNN with lowand high-resolution image pairs constructed in the first step. The input low-resolution image is interpolated to the output size as done in the previous ZSSR method [28]. Then, the interpolated image is mapped to an estimation of the highresolution image. A fully convolutional ResCNN is adopted in our method for estimating the high-resolution image, with four layers, each of which has 64 channels. Besides, ReLU activation is used on the first three layers, while no activation was applied on the last layer. Each layer maintains the original image size. As for the size and number of convolution kernels as done in the previous ZSSR method [28]. Furthermore, L1 loss is used with the Adam optimizer, and the optimization is stopped when we get to a learning rate of 10 −6 .
The third step of the ZSSR method with gradient fusion for THz images is reconstructing high-resolution image with the ResCNN trained in step two. Specifically, a method is adopted for reconstructing the high-resolution image as done in ZSSR method [28] (which generates 8 different outputs for the 8 rotations and flips of the test low-resolution image, and then combines them).
In summary, the difference between the ZSSR method with gradient fusion and the ZSSR method is the dimension of training data and that of super-resolution reconstructed image. The dimension of training data and super-resolution reconstructed image in the ZSSR method is one, but the dimension of training data and super-resolution reconstructed image in the ZSSR method with gradient fusion is three.
As the VDSR method with gradient fusion and the ZSSR method with gradient fusion utilize the gradient maps of training images, the mapping between low-resolution image and high-resolution image is more accurate. In order to quantitatively evaluate the super-resolution performance of the VDSR method with gradient fusion and the ZSSR method with gradient fusion, comparison experiments are addressed in the next section.

V. EXPERIMENTS AND RESULTS
In this section, we selected the VDSR method to evaluate the VDSR method with gradient fusion by comparing their reconstruction performance. In the same way, the ZSSR method was selected to evaluate the ZSSR method with gradient fusion by comparing their performance. Meanwhile, the popular Lucy-Richardson method was selected as a type of deconvolution super-resolution method.
Before performance comparison, we introduced peak signal-to-noise ratio (PSNR) and structural similarity index measurement(SSIM) as the performance index. In the superresolution reconstruction performance comparison experiments, two real datasets are considered. One was collected in 2015 at Universidad de Granada with a passive wave millimeter camera [29], the other was collected in our THz lab with linear scanning THz imaging system.

A. SUPER-RESOLUTION RECONSTRUCITON PERFORMANCE INDEX
PSNR and SSIM are important objective indicators to evaluate the effect of super-resolution method. PSNR is defined as the ratio of peak signal power to average noise power. The calculation formula is expressed as follows.
where x(i, j) denotes pixel (i, j) of the actual high-resolution image and y(i, j) represents pixel (i, j) of the reconstructed high-resolution image. D is the maximum peak-to-peak swing of the signal (255 for 8-bit images). It can be assumed that the noise x(i, j)y(i, j) is uncorrelated with the signal. M and N are the width and height of the actual high-resolution image. SSIM evaluates the similarity between the reconstructed high-resolution image and the actual high-resolution image from the perspective of the image structure. The calculation formula is written as follows.
where µ x and µ y are separately pixel mean of the actual high-resolution image and the reconstructed high-resolution image. σ x and σ y are separately pixel standard deviation of the actual high-resolution image and the reconstructed highresolution image. C 1 and C 2 stand for two constants. It indicates the closer the SSIM value is to 1, the more similar the two images are. The hardware condition of the comparison experiments is CPU intel R Core TM i7-10700K@3.8GHz, 64G memory, and Geforce RTX 2080Ti graphics card, while the software environment is Windows10, Python-3.7, and TensorFlow2.2.

B. SUPER-RESOLUTION PERFORMANCE COMPARISON EXPERIMENT WITH THE PMMWIS
The PMMWIs is collected in 2015 at Universidad de Granada with a passive wave millimeter camera, named  Fig. 4.
Due to nonuniformity of channel in THz scanning imaging system, the THz images have background texture. For a more realistic simulation of the noise situation of THz scanning imaging system, stripe noise is added to each image in the PMMWIs datasets by using (6) [30].
where x ij and y ij denote respectively the response value and the observed value of THz detectors. g i is the gain of the ith THz detector, which is Gaussian distribution (with 1 mean and 0.15 standard deviation), while o i is the bias of the i-th THz detector, which is Gaussian distribution (with 0 mean and 12 standard deviation). In addition, adding stripe noise could enhance the learning ability of neural network and add more edge features for training data. Examples of the THz images are shown in Fig. 4, where Fig. 4(a), Fig. 4(b), Fig. 4(c), and Fig. 4(d) are original THz images in PMMWIs datasets, Fig. 4(e), Fig.  4(f ), Fig. 4(g), and Fig. 4(h) are the example THz images with stripe noise.
In our experiments, we randomly chose 100 images from PMMWIs datasets as the super-resolution reconstructed THz images. Scale factors are 2, 3, and 4. Then, the performance of Lucy-Richardson method, VDSR method, VDSR method with gradient fusion, ZSSR method and ZSSR method with gradient fusion is illustrated in Table 1.
As presented in Table 1, the average PSNR and SSIM values of the Lucy-Richardson method are the least compared with other compared super-resolution methods in scale factor 2, 3, and 4. It means the super-resolution reconstruction performance of the Lucy-Richardson method is the poorest with the compared methods. In scale factor 2, 3, and 4, the average PSNR values of the VDSR method with gradient fusion is improved by 8.14%, 7.54%, and 4.82% compared to that of the VDSR method, respectively. And the average SSIM values of the VDSR method with gradient fusion is improved by 20.02%, 38.41%, and 22.21% compared to that of the VDSR method, respectively. Then we can easily get that the super-resolution performance of VDSR method with gradient fusion is significantly better than that of VDSR method.
Meanwhile, we can notice that the ZSSR method with gradient fusion achieves a significant super-resolution reconstruction performance improvement compared with the ZSSR method in scale factor 2. Although the average SSIM value of the ZSSR method with gradient fusion is improved by 42.53% compared to that of the ZSSR method in scale factor 3, the average SSIM value is decreased from 0.9402 in scale factor 2 to 0.6589 in scale factor 3. It means the super-resolution reconstruction performance decreases as the increase of scale factor. The reason is the limited learning information in low-and high-resolution image pairs for ZSSR method with gradient fusion as the size of the images is small in PMMWIs.
Then we can obtain that the VDSR method with gradient fusion and the ZSSR method with gradient fusion have good super-resolution performance than the VDSR method and ZSSR method separately when the scale factor is 2. When the size of super-resolution reconstructed THz image is small and the scale factor is large, limited learning information could be provided for the ZSSR method with gradient fusion and the ZSSR method. As a result, the performance of the ZSSR method with gradient fusion and the ZSSR method will obviously deteriorate as the increase of scale factor.

C. SUPER-RESOLUTION PERFORMANCE COMPARISON EXPERIMENT WITH OUR THZ LAB DATA
In this part of experiment, THz images are captured by a linear scanning THz camera (Linear Tera-512, detection spectrum range is 50GHz∼0.7THz, pixel number is 512). The THz source's frequency is 0.3THz. The linear scanning THz imaging system is shown in Fig. 5.    In our experiments, four samples are selected, and 20 images are collected for each sample. The size of each THz image is 512×512. The samples and corresponding THz images are shown in Fig. 6.
And we performed the Lucy-Richardson method, the VDSR method, the VDSR method with gradient fusion, the ZSSR method, and the ZSSR method with gradient fusion. Table 2 shows the average PSNR and SSIM values for these super-resolution methods in scale factor 2, 3, and 4.
As can be seen from Table 2, the performance of the Lucy-Richardson method is the worst in scale 2, 3, and 4 (both PSNR and SSIM). Meanwhile, it is observed that the VDSR Examples of reconstruction quality of highlighted windows in the THz images of Fig. 6(b) and Fig. 6(c), zoomed in columns 2-6 constructed by various super resolution methods-(b) the Lucy-Richardson method, (c) the VDSR method, (d) the VDSR method with gradient fusion, (e) the ZSSR method, (f) The ZSSR method with gradient fusion. method with gradient fusion achieves a significant superresolution reconstruction performance improvement compared with VDSR method. In scale factor 2, 3, and 4, the average PSNR value of the VDSR method with gradient fusion is improved by 7.84%, 4.27%, and 4.43% compared to that of the VDSR method, respectively. And the average SSIM value of the VDSR method with gradient fusion is improved by 8.94%, 14.58%, and 15.06% compared to that of the VDSR method, respectively. Likewise, the average PSNR and SSIM values of the ZSSR method with gradient fusion are significantly outperformed to that of ZSSR method in scale factor 2, 3, and 4 as presented in Table 2. This means that in contrast to the VDSR method and the ZSSR method, higher PSNR and SSIM values are achieved in corresponding superresolution methods with gradient fusion.
In order to visually show super-resolution reconstruction effect of the compared methods, the THz images in Fig. 6(b) and Fig. 6(c) were super-resolution reconstructed with scale factor 2, and the super-resolution reconstructed THz images are converted to pseudo-color images. The pseudo-color images of the original THz images are shown in the first column in Fig. 7. To clearly show the super-resolution effect of the compared methods, the same local regions of each reconstructed image were zoomed, and the zoomed pseudo-color images are presented in columns 2-6 in Fig. 7.
In Fig. 7, one of our observations is that the VDSR method with gradient fusion produces relatively sharp edges and high contrast THz image compared with the super-resolution reconstructed result of the VDSR method. In the same way, the ZSSR method with gradient fusion produces slightly sharp edges compared with the ZSSR method in visual. And the THz images reconstructed with the Lucy-Richardson method are the most blurred ones when compared with other super-resolution methods.

VI. CONCLUSION
In this paper, we proposed the gradient degradation model of THz images. The proposed gradient degradation model of THz images accurately describes how a high-resolution THz image's gradient map degrades to corresponding lowresolution image's gradient map. And according to the proposed gradient degradation model of THz images, the low-and high-resolution image pairs' construction method with gradient fusion is proposed for learning-based superresolution methods. Benefit from it, the performance of the VDSR method with gradient fusion is significant improved compared with that of VDSR method. In the same way, the performance of the ZSSR method with gradient fusion is much better than that of the ZSSR method. It means the low-and high-resolution image pairs' construction method with gradient fusion could effectively improve the performance of the learning-based super-resolution methods without gradient fusion. And the results of our comparison experiments validated this. For other learningbased super-resolution methods, such as the enhanced deep residual networks for single image super-resolution (EDSR) and the second-order attention network for single image super-resolution (SAN), it is easy to apply the proposed low-and high-resolution image pairs' construction method to improve super-resolution reconstruction performance.
In our work, we added the gradient information to training pairs and improved super-resolution performance. Further more, we could try to extract gradient map's features with a separate neural network and fuse the extracted features with those extracted from original training images. Because an separate neural network for edge feature extraction is more flexible and accurate compared to neural networks applied in this paper. Thus, this will be researched in our future work.