Finger Vein Image Inpainting With Gabor Texture Constraints

The texture edge continuity of a finger vein image is very important for the accuracy of feature extraction. However, the traditional inpainting methods which, without accurate texture constraints, are easy to cause the vein texture of the inpainted image to be blurred and break. A finger vein image inpainting method with Gabor texture constraints is proposed. The proposed method effectively protects the texture edge continuity of the inpainted image. Firstly, using the proposed vertical phase difference coding method, the Gabor texture feature matrix of the finger vein image, which can accurately describe the texture information, can be extracted from the Gabor filtering responses. Then, according to the local texture continuity of the finger vein image, the known pixels, which have different texture orientations with the center pixel in the patch, are filtered out using the Gabor texture constraining mechanism during the inpainting process. The proposed method eliminates irrelevant information interference in the inpainting process and has a more precise texture propagation. Simulation experiments of artificially synthetic images and acquired images show that the finger vein images inpainted by the proposed method have better texture continuity and higher image quality than the traditional methods which do not have accurate texture constraints. The proposed method improves the recognition performance of the finger vein identification system with the acquired damaged images.


I. INTRODUCTION
Compared with other biometrics, finger vein [1], [2] has these advantages: non-contact, internal characteristics, living body recognition, high level of security, etc. The recognition performance of a finger vein identification system is very dependent on the image quality [3]. However, due to the multiple scattering interaction in biological tissue, the captured finger vein images are sometimes degraded, resulting in blurring of some vein regions. Reference [4] proposed a bilayer restoration model and [5] proposed a simple and effective scattering removal model, which effectively solve the degradation of finger vein images caused by tissue scattering and improve the visibility of the images. Nonetheless, these methods can be used to restore the degraded image when the image is unpolluted. However, the pollution on The associate editor coordinating the review of this manuscript and approving it for publication was Vishal Srivastava. the mirror of iacquisition equipment or the peeling finger as shown in Fig. 1 will make a part of the acquired image to be covered, which leads to the loss of image information. The damaged regions will increase the difficulty of finger vein feature extraction and reduce the recognition performance. VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ Therefore, the research on finger vein image inpainting has important theoretical and practical significance. However, the characteristics of finger vein images with different width and extension orientations of vein branches [6] and weak texture edge make that work more difficult. Many image inpainting methods can be classified into two main categories [7], [8]: structure-based and texturebased. Structure-based inpainting methods [9]- [15] calculate the gradient field or second derivative field of the image and then diffuse information by isophotes from the known regions to the unknown regions point-by-point. Most of the structure-based inpainting methods [9]- [14] are also known as partial differential equations (PDE) based methods or total variational (TV) based methods. These methods ensure local intensity smoothness so that they can inpaint small regions well generally. However, this kind of method essentially ignores the importance of texture information for finger vein images, resulting in blurring when inpainting the vein texture regions. In addition, [15] proposed a fast marching method (FMM). FMM determines the priority by estimating the direction isophotes and then uses the neighborhood pixels for weighting average to fill the damaged region. FMM improves the time efficiency compared to PDE-based methods, but it is blind to use all known information in the patch for weighting calculation, which leads to that the vein texture edge can not be maintained.
Texture-based inpainting [16]- [18] is capable of inpainting larger holes by copying and pasting similar patches in the image using a searching strategy. For example, Criminisi method [16], which combines the advantages of texture synthesis and diffusion filling. However, the priority of the patch is easy to be disordered in the Criminisi method, which leads to the blocky effect in the inpainted image. In recent years, scholars have also proposed many improved methods. Reference [17] improved the Criminisi method by using image structure tensors so that the impact of the image structures is strengthened during the inpainting process. The proposed method tackles two limitations of the image structure propagation and the filling-in order in the image inpainting process. Reference [18] uses a robust priority function to avoid a dropping effect and region segmentation to determine the adaptive patch size and reduced search region.
Recently, there have been some inpainting methods, which combine structure and texture information [19]- [21]. Bertalmio et al. [20] proposed an inpainting method combining texture synthesis and PDE. In this method, the image is decomposed into texture and structure components. Then reconstruct each of them separately with structure and texture filling-in algorithms to complete the inpainting task. Reference [21] proposed a PDE-based image inpainting method using an anisotropic heat transfer model, which can simultaneously propagate the structure and texture information. Although this kind of method combines the advantages of the two, it increases the complexity of the method and has limited ability to improve the performance in finger vein image inpainting.
The Criminisi method and those described above improved methods select the best matching patch by the sum of square difference (SSD) distance criterion or other grayscale information of the patch, which can not accurately describe the texture of finger vein images. It may lead to matching errors and image information loss when inpainting the vein texture regions of the finger vein image.
Recently, with the rapid development of deep learning, it has achieved good results in such fields as finger vein image process and recognition. Reference [22] proposed a convolutional neural network (CNN) to train and restore the vein patterns, which tackles the problem of vein pattern loss caused by overexposure or blurred region in the finger vein images. Reference [23] proposed a new model based on the pulse coupled neural network (PCNN) to enhance finger vein image quality and further to improve the reliability of image recognition. Reference [24] applied CNN to finger vein recognition and achieved better performance than traditional algorithms. However, their research topics are based on the unpolluted finger vein images. There is a lack of deep learning research on the inpainting methods of the damaged finger vein images.
In this paper, a finger vein image inpainting method with Gabor texture constraints is proposed to tackle the texture edge discontinuity of the inpainted finger vein image. First, the Gabor texture feature matrix (GTFM) of the finger vein image is extracted from Gabor filter responses using a proposed vertical phase difference coding method. The GTFM can accurately describe the texture information of the finger vein image. Then, the known pixels, which have different texture orientations with the center pixel in the patch, will be filtered out by the Gabor texture constraining mechanism during the inpainting process. The Gabor texture constraining mechanism makes the proposed method has a more precise texture propagation. The vein texture of the inpainted finger vein image is more complete and coherent than the traditional inpainting methods which do not have accurate texture constraints.
The rest of the paper is organized as follows. Section II presents a vertical phase difference coding method based on Gabor filter. Section III presents a finger vein image inpainting method given Gabor texture constraints. The experimental results and analysis are given in Section IV. Section V gives a summary of the paper.

II. VERTICAL PHASE DIFFERENCE CODING METHOD BASED ON GABOR FILTER
Two-dimensional Gabor filter is very sensitive to the texture edge of the image and its expression of frequency and orientation is consistent with the texture recognition mechanism of the human vision system [25]. Gabor filter can effectively enhance the edge, peak, valley, ridge contours and other underlying feature information. Moreover, the Gabor filter has good scale characteristic and directivity, which can match the characteristics of finger vein images, such as different width and extension orientation of vein branches. And the Gaussian kernel component of the Gabor filter can be robust to noise interference. So compared to the gradient operator which can be greatly disturbed by light intensity and noises, Gabor filter can extract texture edge of finger vein images more stably and accurately.

A. EXTRACT TEXTURE EDGE BY GABOR FILTER BANK
The Gabor filter is a Gaussian function windowed by a sine plane wave, which is expressed as follows [26], [27]: where x = x cos θ + y sin θ, y = −x sin θ + y cos θ; λ and ψ are the wavelength and the phase offset of sine wave function, respectively; σ is Gaussian standard deviation, also known as the spatial scale factor; γ is the spatial aspect ratio of the Gabor filter. The parameters are related to the size of images used in this paper, 200 × 84. Thus, set λ = 17, ψ = 0, γ = 1, and the three scales are σ 1 = 3, σ 2 = 3.4, σ 3 = 3.8. θ is orientation parameter that determines the directivity of Gabor filter, which can be expressed as follows: where N is the total number of orientations. According to the characteristics of vein structure, we set N = 8 after a lot of experiments. θ k represents the k th orientation, π/N is the angular interval. The real component g re and the imaginary component g im of the Gabor filter can be obtained by decomposing the sine wave part of Eq. 1: After convoluting them with the original finger vein image F(x, y), the coefficient amplitude can be obtained by taking the modulo according to , are the filter responses of the real component and the imaginary component at k th orientation and σ scale respectively. H k,σ (x, y) is the coefficient amplitude, which represents the energy of images in that orientations and scales as shown in Fig. 2.
The larger the coefficient amplitude is, the richer the texture edge information is saved on that scale. Thus, firstly, we select the largest of the three scales as the amplitude coefficient of each point in that orientation: The responses H k (k = 1, 2, . . . , 8) in 8 orientations are obtained by using Eq. 6.

B. VERTICAL PHASE DIFFERENCE CODING METHOD
According to the principle that the texture edge information difference between the two orientations along the finger vein and vertical to the finger vein is the largest, a vertical phase difference coding method is proposed in this paper. The filtering responses H k in 8 orientations are divided into 4 groups (H 1 and H 5 , H 2 and H 6 , H 3 and H 7 , H 4 and H 8 ) according to the mutual vertical orientation. After calculating the absolute value of the difference of each group, the group with the largest result will be picked as follows where k ∈ [1,4], the orientations k max and k max ⊥ of the group H k max are the possible texture orientations of the pixel to be repaired. The larger the response amplitude is, the stronger the texture edge information energy is in that orientation. Therefore, the Gabor texture feature matrix of the finger vein image can be finally obtained by using the following equation where H k max and H k max ⊥ are the Gabor filtering responses at the k max direction and the k max ⊥ direction respectively. Fig. 3 shows the test image and the corresponding GTFM. Combining Fig. 3(a) and Fig. 3(b) shows that the GTFM of finger vein image clearly indicates the texture trend of the vein regions and the background regions, including some texture edge of slight veins. In addition, the texture feature distribution of the image blocks located on the same vein structure has a high degree of similarity. These characteristics demonstrate that GTFM accurately describes the texture information of the finger vein image. Thus GTFM can be used for auxiliary correction to protect the vein texture edge of the finger vein image during the inpainting process.
As for the damaged finger vein images, the convolution window on the contour of the damaged border contains unknown pixels. The grayscale values of the unknown pixels in the window are set to zero so that those pixels do not affect the accuracy of convolution calculation. Finally, the known pixels are encoded and assigned texture feature values, while the texture feature value of the unknown pixels (damaged pixels) is set to zero, according to

III. IMAGE INPAINTING WITH GABOR TEXTURE CONSTRAINTS
As for the traditional methods which do not have accurate texture constraints, due to mixing low texture correlation information in the filling calculation, the vein texture of the inpainted image will be discontinuous. The proposed method utilizes GTFM to constrain the texture propagation in the inpainting process. The specific steps of the proposed method are as follows.

A. DETERMINATION OF PRIORITY
Priority is an important factor affecting the performance of image inpainting methods. Sethian [28] proposed fast-marching level-set methods to track the moving edge, which can simulate the curve evolution process of the contour of a damaged region. It starts from the contour and gradually spreads to the interior, point by point, until all unknown pixels are filled. It ensures that the filling sequence is more in line with the human visual psychology. The time of the contour passes through each pixel can be expressed as T (i, j) and the priority of the proposed inpainting method can be determined by T (i, j). The pixels with the smallest T (i, j) value will be filled first. The diffusion of the contour satisfies the conditions of the Eikonal equation [28]: where v i,j is the diffusion speed of the contour of the damaged region and v i,j = 1 means moving one pixel at a time. Using the inverse difference method to solve Eq. 10, we can get T (i, j) of each unknown pixel: i,j T are the forward and backward differences of the time function T in the horizontal x orientation and the vertical y orientation respectively. The detailed calculation process can be found in [28]. Fig. 4(b) and Fig. 4(c) show the Gabor texture feature information in the patch centered on damaged pixels A and B, respectively. The pixel A is located in the vein region, while the pixel B is located in the background region. The feature values of the unknown pixels are zero and other non-zero values are the known pixels. Combining the observations in Fig. 4(a) to 4(c) shows again that the texture trends of the vein regions or the background regions in the local image are continuous. Those pixels with the same texture trend have the same texture feature value. Therefore, we select the mode of the Gabor texture feature value of all known pixels in the patching window as the main texture feature of that window.

B. DETERMINATION AND UPDATE OF THE MAIN TEXTURE ORIENTATION
As shown in Fig. 5, B ε (p) is the patch centered on pixel p and its diameter is ε. The known regions and the unknown regions can be determined by the mask binary image, in which the value of pixels in the known region is 1, while that in the unknown region is 0. And the mask binary image can be obtained by the fuzzy c-means (FCM) segmentation algorithm [29]. GTFM B ε (p)| known represents the Gabor texture feature of all known pixels in B ε (p). Then the texture feature value of p is updated as follows: where mto B ε (p) is the main texture orientation of B ε (p). Too small a patch may lead to too little known pixels being available during the inpainting, and too large a patch may introduce too much noise, affecting the determination of the main texture orientation. The size of the patch is set to 7 × 7 after a lot of experiments. As a result, the texture feature of damaged pixels A and B in Fig. 4(a) will be updated to 2 and 7 after filling, respectively. It is in line with the texture trend of the finger vein image.

C. GABOR TEXTURE CONSTRAINING MECHANISM AND WEIGHT COEFFICIENT OF INPAINTING
Considering that texture continuity is very important for the inpainted finger vein image, the proposed method gives the priority to the texture correlation between the pixels during inpainting. Firstly, according to the main texture direction obtained in Eq.12, the proposed method uses the Gabor texture constraining mechanism to eliminate the interference of information with different texture orientations in the filling calculation. The Gabor texture constraining mechanism enables the proposed method achieve a more precise texture propagation. Secondly, according to the grayscale and distance correlation between the pixels, the proposed method inpaints the damaged finger vein image by weighting the remaining known pixels which are in the same texture direction. In Fig. 5, the pixels in the dotted box part outside are known. Those known pixels will be used to fill the unknown pixel p. Before this, the known pixels that have different texture orientations with the center point will be filtered out by the Gabor texture constraining mechanism as shown in Fig. 6.   6 shows the distribution of the Gabor texture features in the patch and different colors represent different texture direction features. The white points and the center point of the patch are the damaged pixels whose information is unknown and the other color points (except the center point) are known. The texture feature value of the center point has been assigned by the main texture orientation of that patch. The green points and the center point are located on the same vein texture trend and have the same texture orientation, while the other non-green points have different texture orientations with the center point. The proposed method will filter out the points with different texture orientations and select the green points for weighting calculations. The Gabor texture constraining mechanism makes the inpainting process free from the 'noises' with low texture correlation, which plays a role in maintaining the vein texture edges and avoiding blurring the inpainted region. Fig. 7(a) to 7(c) are the GTFM extracted from the original image, the damaged image (obtained by manually adding a damaged region to the original image), and the inpainted image (obtained by inpainting the damaged image by the proposed method) respectively. It can be seen that the GTFM of the inpainted image restores the vein texture trend of the original image, which demonstrates that the Gabor texture constraining mechanism can constrain the inpainting process accurately following the texture trend. i At the same time, considering the spatial and the grayscale correlation between the pixels of the finger vein image, VOLUME 8, 2020 the proposed method sets the distance factor and the gray factor to weight the remaining known pixels as follows: 1) Firstly, calculate the average grayscale avg of the remaining known pixels. Then the absolute values of the difference between q and avg, and the Euclidean distance between q and the center pixel p can be calculated as follows where x ∈ B ε (p)| known , q is one of the remaining known pixels. gray is the gray factor. The pixels whose grayscale value is closer to avg get greater weight. dst is the distance factor. The pixels which are closer to the center pixel p get greater weight.
2) The weight of each remaining known pixel q is obtained by using the exponential component of the standard normal distribution model: 3) Weights normalization: 4) The final weighting equation is defined as follows: whereF(p) is the value of pixel p after inpainting. Fig. 8 shows the flowchart of the proposed method and the specific steps are as follows:

D. THE OVERALL FLOW OF PROPOSED METHOD
(1) Firstly, the proposed method uses the fuzzy c-means (FCM) segmentation algorithm [29] to get the mask binary image and then extract the GTFM of the finger vein image by the vertical phase difference coding method proposed in Section II.
(2) Using a fast-marching level-set method, the time value of the damaged pixels on the contour of the damaged region can be obtained to determine the priority.
(3) The main texture orientation of the patch can be determined by GTFM. Then the method uses the Gabor texture constraining mechanism to filter out the known pixels, which have different texture orientations from the main texture orientation in the patch.
(4) The remaining known pixels are used for weighting calculation to fill the damaged pixel. Then the time value and the Gabor texture feature of the inpainted pixel will be updated.

IV. EXPERIMENT RESULTS AND EVALUATION
The proposed method is compared with three traditional inpainting methods by experiments on the artificially synthetic finger vein images and the acquired finger vein images, respectively. The iteration times of the method based on TV model [9] are set to 1000. The patch size of the Crinimisi method [16] and the FMM method [15] are the same as that of the proposed method, which is set to 7 × 7. Besides the subjective evaluation of the human visual system, mean square error (MSE) and peak signal to noise ratio (PSNR) are used to evaluate the inpainting quality of the artificially synthetic damaged images. False rejection rate (FRR) and false acceptance rate (FAR) are also used to evaluate the inpainting quality of the acquired damaged images.

A. ARTIFICIALLY SYNTHETIC DAMAGED IMAGE
As shown in Fig. 9, the artificially synthetic damaged image Fig. 9(b) was obtained by adding four damaged regions to the test image Fig. 9(a). The location of each region is different: A horizontally covers a wider vein; B is located in a smooth background region; C covers a vein vertically; D covers a slight vein with blurred texture. The types of damaged images in practical applications are mostly included in these cases. Fig. 10(a) to 10(d) show the results of three traditional inpainting methods and the proposed method and the inpainted regions are enhanced for the convenience of observation. It can be seen that for the damaged region A and C, where the veins are masked, the vein texture of the result Fig. 10(a) inpainted by the method based on TV model is blurred and incoherent and the vein is cut off visually; The result Fig. 10(b) inpainted by the Crinimisi method has an obvious blocky effect, which affects the edge of a vein; In the result Fig. 10(c) of the FMM method, the vein edge was blurred or the boundary of the inpainted region was obvious, and the effect of edge retention is unsatisfactory. For the damaged region B located in the smooth background region, all these methods work well. As for the damaged region D with low contrast, the vein structure in this region is slight and blurred. The method based on TV model or FMM caused blur and rupture to the vein structure, while the Crinimisi method and the proposed method do better in inpainting slight veins. As shown in Fig. 10(d), due to the Gabor texture constraining mechanism, the proposed method has a precise texture propagation and the image inpainted by the proposed method has a better effect on vein edge retention and visual connectivity.

2) OBJECTIVE EVALUATION AND ANALYSIS
The comparison of the performance parameters in Table 1 shows that the proposed method has higher image quality than the other three traditional methods without accurate texture constraints.

B. ACQUIRED DAMAGED IMAGE
The above experiments have demonstrated the effectiveness of the proposed method for the artificially synthetic damaged images. The research in this part will be performed on the acquired damaged images. Because of the vein images in the current public databases are all acquired under normal conditions, such special case images are lacking. Thus, the databases used in this part consist of the images acquired under damaged conditions and the same users under normal conditions, by a laboratory at Hangzhou Dianzi University (HDU).
The damaged image includes two conditions: the mirror of the acquisition equipment is polluted or the fingers are peeling. A total of 40 fingers of 20 volunteers (15 men and 5 women), whose fingers have peeled, were acquired. Under normal condition and two kinds of damage conditions, 10 images are collected for each finger respectively, 40 × 10 × 3 = 1200 in total. These acquired images constitute a normal finger vein image database named NFVI and two damaged finger vein image databases named DFVI. Then, the NFVI and two other DFVIs are respectively composed into two mixed finger vein image databases named MFVI, each finger class in MFVI contains 10 normal images and 10 damaged images. For the acquired damaged images, the original images cannot be obtained, so it is no longer meaningful to use MSE and PSNR for evaluation. Therefore, the FRR and FAR of intra-class and inter-class samples are used to verify the effectiveness of inpainting methods. After the damaged images in the MFVIs being inpainted by different inpainting methods, all of the images in the MFVIs will be processed by the guide filter enhancement method [30], the Niblack [31] segmentation method, and the Zhang-Suen thining method [32] to obtain the finger vein skeleton features. Finally, the modified Hausdorff distance (MHD) [33] method is used for recognition and each MFVI can obtain 7,600 intra-class matching data and 312,000 interclass matching data. The recognition performance comparison is shown in Table 2.

1) MFVI WITH IMAGES OF PEELING FINGERS
It can be seen from Fig. 11(b) that, when the user's finger is peeling, there will be a slender strip-shaped damaged region in the acquired image. The damaged region is similar to the vein structure in grayscale and covers a part of the vein structure, which causes the vein structure in this region to break in the skeleton. In the result Fig. 11(c) inpainted by the Crinimisi method, the blocky effect existing in the inpainted region can be clearly seen and that region in the corresponding skeleton is also disordered. From the result Fig. 11(d) of the FMM method, the vein edge becomes blurred after being inpainted, which causes the vein of that region in the skeleton to break. From the result Fig. 11(e) inpainted by the method based on the TV model, it can be seen that the connection of the vein structure is incoherent, causing a false skeleton trend in that region. While inpainted by the proposed method, both the result and the skeleton in Fig. 11(f) are more in line with the vein texture structure of the image acquired under normal conditions.
The receiver operating characteristic (ROC) curve [34] in Fig. 12 shows that such a peeling region seriously affects the grayscale distribution of the image, which leads to inaccurate segmentation and reduces the recognition performance. Combined with Table 2, it can be seen that, when the FAR is 0, the FRR of the proposed method is 23.25%, TV model is 24.39%, FMM method is 27.43%, Crinimisi method is 28.6%, while without any inpainting method is 31.51%, respectively. It shows the best recognition performance after being inpainted by the proposed method. Fig. 13(b) shows another type of damaged image in practical applications. When there is a dirty block on the mirror of the acquisition equipment, a black block region will appear on the acquired image, resulting in the loss of image information in this region. Although the black block has been removed in the result Fig. 13(c) of the method based on TV model, the visual continuity is poor and the skeleton is not consistent with the original vein trend. In the result Fig. 13(d) of the FMM method, there are certain obvious repair boundaries in that region, which leads to the false trend of the vein skeleton. In the result Fig. 13(e) of the Crinimisi method, the edges of the veins are blurred, resulting in a false skeleton. The proposed method uses the GTFM to constrain texture propagation, which maintains the texture edges of the inpainted result Fig. 13(f) better, so that the skeleton maintains the same vein structure as the image acquired under normal conditions. The ROC curve in Fig. 14 shows that, if such a block damaged region exists in the finger vein image, the original vein structure will be masked and the recognition performance will be reduced. Combined with Table 2, it can be seen that, when the FAR is 0, the FRR of the proposed method is 20.12%, Crinimisi method is 22.12%, FMM method is 23.10%, TV model is 23.54%, while without any inpainting method is 28.91%, respectively. It shows the best recognition performance after being inpainted by the proposed method. The analysis of the above experiments demonstrate that both for the artificially synthetic damaged images and two kinds of acquired damaged images, the proposed method performs better compared with the other three traditional methods. Due to using the Gabor texture constraining mechanism during the inpainting process, the vein texture edge of the images inpainted by the proposed method is more continuous. As a result, the vein skeleton features are more accurate and the recognition performance is also better than three traditional methods without accurate texture constraints.

V. CONCLUSION
This paper proposes a finger vein image inpainting method with Gabor texture constraints, which makes full use of texture information to constrain texture propagation during the inpainting process. The Gabor texture constraining mechanism ensures that the proposed method has a more precise texture propagation so that the vein texture continuity of the inpainted image is better. The proposed method effectively overcomes the problems that the traditional inpainting methods, without accurate texture constraints, easily cause the vein texture edge of the inpainted finger vein images to be blurred and break. Simulation experiments of artificially synthetic images and acquired images demonstrate that the images inpainted by the proposed method have better texture edge continuity and higher quality than traditional image inpainting methods. The proposed method improves the recognition performance of the finger vein identification system for damaged images.