Low Dimensional Manifold Regularization Based Blind Image Inpainting and Non-Uniform Impulse Noise Recovery

Blind image inpainting is a challenging task in image processing. Motivated by the excellent performance of low dimensional manifold model (LDMM) in image inpainting for large-scale pixels missing, we introduce a novel blind inpainting model to repair images with missing pixels or damaged with impulse noise, in spite of the unknown locations of the corrupted pixels. We applied logarithmic transformation to separate the image and binary mask. LDMM regularization and $l_{0}$ norm regularization were applied respectively to simultaneously estimate the image and the mask. The resulted minimization problem was then solved by the split Bregman algorithm. The simulation results showed that the proposed model, compared with the existing ones, can effectively restore the image with large uniform and non-uniform missing rate.


I. INTRODUCTION
Image inpainting is a very important problem in image processing. It has been widely studied and applied in many fields, such as digital restoration [1], art conservation [2], and video inpainting [3]. Therefore, image inpainting has attracted much attention. A variety of image inpainting methods have been developed to restore the image in past decades. Up to now, image inpainting methods can be roughly classified into two categories, the knowledge-driven methods and the data-driven methods. The knowledge-driven methods and the data-driven methods are two different approaches, and they are all being studied in parallel.
The PDE-based methods are the most fundamental methods of all image inpainting methods. The basic idea is to extend effective information in the area surrounding the The associate editor coordinating the review of this manuscript and approving it for publication was Sudipta Roy . corrupted region so as to achieve the purpose of restoration [13]. In [13], Bertalmio et al. first proposed the concept of digital image inpainting and put forward an inpainting model based on third-order PDE (BSCB model). Chan et al. proposed the total variation (TV) model [14]. As a further improvement of the TV method, the curvaturedriven diffusions (CDD) inpainting model was developed by Chan et al. [15]. Typical models of the PDE-based methods also include Mumford-Shah model [16], Euler's elastica model [17], Mumford-Shah-Euler model [18], etc.. These methods can achieve good results for image inpainting of small scale damage, e.g. removing scratches, removing text coverage, filling holes, and so on. However, they often result in blurring of structure or texture for image inpainting of large scale damage, such as target removal.
The basic idea of the exemplar-based methods is to fill in the corrupted region by copying the patch from the source regions. The most representative method is proposed by Criminisi et al. [19], which guides the inpainting order by defining the priority of patch, so that texture and structure information can be propagated simultaneously. Compared with the PDE-based methods, the exemplar-based methods are not only used to effectively repair large-scale damage, but also to improve the restoration efficiency and guarantee the integrity and continuity of edge structure. However, some limitations can still be found through these methods, such as block mismatch, and unreasonable repair order.
The basic idea of the SR-based methods is to restore the damaged image using the over-complete dictionary and sparse coding. In 2005, Elad et al. [20] proposed an inpainting method based on Morphological Component Analysis (MCA), which can simultaneously realize the restoration of cartoon layer and texture layer. In 2006, Elad et al. [21] proposed a K-SVD algorithm, and then used it to repair the missing pixels in images. Although good visual effects for smooth images can be found through these methods, they still have some problems, e.g. time-consuming learning, high computational complexity, very limited prior knowledge, and poor adaptive ability.
The second category is the data-driven methods, i.e. deep learning-based methods [22]- [24]. The main idea is to use large amount of real images for training and learning, so as to automatically repair the damaged regions of the image and achieve the purpose of image restoration. Köhler et al. [25] established a deep learning framework based on the multilayer perception (MLP) architecture for image inpainting. Zhu et al. [26] proposed a new technique depending on CNN, which assisted in detecting the patch-oriented inpainting process. Zeng et al. [27] designed a Pyramid-context Encoder Network (PEN-Net) for image inpainting. Jiang et al. [28] proposed an inpainting model based on generative adversarial networks (GAN), which contained a generator, a global discriminator, and a local discriminator, and achieved more realistic restoration results.
In most literatures, the inpainting region is assumed to be given by users. However, the corrupted region may not be readily available in some practical applications. In other words, both the original image and the inpainting region are unknown. e.g., removing certain scratches from archived photographs. We call such an inpainting problem a blind inpainting problem, which is an ill-posed inverse problem.
Recently, some work has been done on solving such blind inpainting problem. Ji et al. [29] proposed an approach of l 1 norm of tight frame coefficients to restore the image with missing pixels. In both cases the l 1 norm was used as the fidelity measurement to suppress the outlier effect of damaged pixels. On the basis of this work, Dong et al. [30] proposed a method by which the l 1 norm of wavelet frame coefficients of images were used as the regularization term. Moreover, a l 0 norm and TV regularization approach is proposed [31] to simultaneously estimate the image and the mask.
In recent years, low dimensional manifold model (LDMM) has been applied to many image restoration tasks [32]- [34], and it has the best results among all data-driven-based image inpainting methods for large-scale pixels missing. Therefore, we used LDMM as the regularization term for images in our approach. The goal of this paper is to develop a model which estimates both the image and the mask for solving such blind image inpainting problem.
In this paper, we propose a novel blind image inpainting method based on LDMM to tackle such an ill-posed inverse problem. Firstly, we formulate the masking operation as a summation after logarithmic compression. Then, we apply a LDMM regularizer and a l 0 norm regularizer on the term corresponding to the logarithm of the image and the mask, respectively. Our proposed model can be solved iteratively using the split Bregman method [35].
Compared with the state-of-the-art inpainting methods, the main contribution of this paper lies in two aspects, one is blind inpainting under LDMM regularization, and the other is the non-uniform impulse noise removal.
The paper organized as follows. In Section II, we introduce LDMM briefly. In Section III, we propose the blind image inpainting model based on LDMM. The split Bregman iteration based algorithms are applied to solve the optimization problems resulted from the proposed model. Numerical experiments are conducted in Section IV for three image restoration tasks: 1. blind image inpainting with different percentage of uniform pixels missing, 2. impulse noise removal with different percentage of non-uniform pixels corrupted. 3. blind inpainting for RGB color image. Finally, a brief conclusion will be given in Section V.

II. LOW DIMENSIONAL MANIFOLD MODEL
As for LDMM, it uses the patch manifold dimension associated with the image as a regularization to restore the image in that the patch manifold dimension associated with many natural images have low dimension.
Consider an m×n size discrete grayscale image I ∈ R m×n . For any pixel x = (i, j), where 1 ≤ i ≤ m, 1 ≤ j ≤ n, let this pixel constitute the left-top pixel in an 2D rectangle patch of size s 1 × s 2 of the image I , and denote this patch as (PI )(x). Define S(PI ) as the collection of all such patches, i.e.: where is an index set to make the union of the patch set S(PI ) cover the whole image, the boundary points are using reflection extension in this paper. The patch set S(PI ) can be seen as a point cloud in R d , where d = s 1 × s 2 . It is observed that S(PI ) samples a low dimensional smooth manifold M (I ) embedded in R d , which is called the patch manifold associated with image I . Based on the assumption that the patch manifold of many natural images has low dimensional structure, Osher et al. [32] proposed LDMM in which the dimension of the patch manifold was used as a regularization to recover an image: where dim(M ) is the dimension of the manifold M .
In LDMM [32], the dimension of the patch manifold M is computed as follows: where α i (i = 1, 2, · · · , d) are the coordinate functions on M , i.e., Using the above formula, the problem (4) can be rewritten as where γ > 0 in the penalty term is a parameter. The above optimization problem (5) of highly nonlinear and non-convex is solved by iterative method. The most difficult part of iterative computing is to solve the following type of optimization problem: where r can be any α i , s(y) is a given function on S, and β > 0 is a penalty parameter.
For (6), with a standard variational approach, the key step problem is to solve a Laplace-Beltrami equation on a point cloud: where ∂M is the boundary of M and n is the out normal of ∂M . If M has no boundary, ∂M = ϕ. Equation (7) is solved by the point integral method (PIM) [36], [37]. It is discretized as follows: where δ > 0 is a parameter, s(y) is the given value. For a parameter t > 0, where C t is the normalizing factor, If we set R(z) = exp(−z), thenR t = R t are Gaussians. The LDMM, with the PIM used, has been shown to achieve good performances, especially in non-blind image inpainting.

III. PROPOSED MODEL AND NUMERICAL ALGORITHM A. OUR PROPOSED MODEL
In this subsection, we apply LDMM to blind image inpainting. We first give a random pixel missing model under additive Gaussian noise, and it can also represent the problem of multiplicative impulse noise under additive Gaussian noise. In our problem, we need to simultaneously estimate the image I ∈ R b and the mask A ∈ R b×b . The diagonal element of the mask matrix A is 1, when a pixel is observed, or the diagonal element of the mask matrix A is 0, when a pixel is corrupted or missing pixel. The problem of estimating I and A, reported in [31], is where E is identity matrix, θ 1 , θ 2 > 0 are the regularization parameters. If the element of the diag(E−A) is 0, it means that the pixel is observed, and if it is 1, it means that pixel is the noise pixel. In this paper, our goal is to estimate the image I from the partial observations I 0 without prior knowledge of the observation mask A. Since I and A are multiplied, the equation (11) is difficult to solve.
There are a large number of methods to remove additive Gaussian noise, but we ignore the influence of additive noise to simplify the model and only discuss the problem of random pixel loss or multiplicative impulse noise. In order to separate I and A, We use a logarithmic transform on both. Firstly, we consider diagonal matrix A = diag(a) with a ∈ { 0, 1 } b . When a pixel with index k is observed, the corresponding mask element a k = 1, and when pixel k is lost, a k = 0. Suppose that we don't take the noise case into account, a pixel k in image I is defined as the scalar product, Secondly, defining g k = log a k , we have where K is a positive integer and K > log 255. To avoid taking a logarithm of 0, we make such an approximation. We take the l 0 norm constraint, so the effect of different K values on the result is the same. Then, assuming that I , A are always positive, we apply a logarithmic transform on (12), here f k = log j k , y k = log(i k + ς ). i.e. y = f + g . A small positive bias term ς > 0 is added to y k to guarantee positivity. Reorganize the data in y , f , g ∈ R b , and return it to y, f , g ∈ R m×n , where b = m × n, i.e. y = f + g. Thus, our task is estimating f and g, given the log transformed observation y. Previously, TV regularization on log transformed images has been used [38], and synthesis models which provide enhanced sparse representations in transform domains have also been used for image denoising and restoration [21]. We assume that logarithmic transformation f of our image I is piece-wise smooth. Meanwhile, the negative elements of g correspond to the non-observed pixels, and the zero elements of g correspond to the observed pixels. Therefore, we take the dimension of the patch manifold as a regularization on the log transformed image f , and the l 0 norm as a regularization on the log transformed mask g. We propose the following minimization model: where λ 1 , λ 2 > 0 are the regularization parameters respectively. The dimension of the patch manifold M is computed as where α i (i = 1, 2, · · · , d) are the coordinate functions on the manifold M . The optimization problem (17) can be reformulated as

B. NUMERICAL ALGORITHM
In this subsection numerical algorithm is presented to solve the proposed model (18). Since the problem (18) is highly nonlinear and non-convex, our algorithm is built upon the split Bregman iteration [39]. To solve the model (18), we adopt the iterative schemes as follows: First, the manifold is fixed, f , g, and the coordinate functions are computed. With a guess of the manifold M n , a guess of g n and a guess of f n satisfying S(Pf n ) ⊂ M n , we use the idea of the split Bregman iteration to update f , g, and α i sequentially.
• Repeat above two steps until convergence. The equation (19) can be solved using PIM, problem (20) has a solution given by solving a least-squares problem, and (21) can be computed using the hard threshold. Thus, we obtain the complete description of the algorithm for solving (18) in Algorithm 1.
where H √ λ 2 (·) is the hard threshold operator and is defined as, And finally, we getf andĝ as estimates of f and g, respectively. Reorganize the data inf ,ĝ ∈ R m×n , and return it tof ,ĝ ∈ R b . The estimates of the image and the mask are computed by inverting the logarithmic transformation, i.e. I = 10f ,Â = 10ĝ .

IV. EXPERIMENTS RESULT
In this section, we carry out experiments on synthetic images to demonstrate the performance of our method, with comparison to the representative method: BITV [31] method code can be found at (https://github.com/manyaafonso/Blind-Inpainting-l0-TV), KALS [40] method code can be obtained Algorithm 1 Algorithm for Solving the Model (18) Require: Initial guess of the image f 0 , g 0 , d 0 = 0. Ensure: Restore f , g. 1: while not converge do 2: Compute the weight matrix W = (w) ij from S(Pf n ), where i, j = 1, · · · , T and T = |S(Pf n )| is the total number of points in S(Pf n ), And assemble the matrices L, W , andW as follows: 3: Solve the following linear systems: where V = S(Pf n ) − d n . 4: Update f by solving a least-squares problem:   16GHz) and 4GB memory. The usual pixels missing recovery problem requires the assumption that the pixels missing location is known and we are blind here, and the usual multiplicative impulse noise problem assumes that the noise distribution is uniform and we can be non-uniform here.
To quantify the quality of the inpainting results, we use the peak signal-to-noise ratio (PSNR) [41] as the measurement indexes, which are defined as follows:  In addition, the structural similarity (SSIM) [42] is introduced to evaluate the similarity between I and I o . The high PSNR and SSIM values of the image can show good performance of the model.
During the experiment of our model, there are two important parameters (λ 1 , λ 2 ) to be tuned. We found that λ 1 = 0.5, λ 2 = 0.1 can provide optimal results of our method in blind inpainting. For BITV and KALS methods, the parameters and the stopping criterion were tuned mutually to achieve the maximal PSNR or the best SSIM for a fair comparison.

A. BLIND IMAGE INPAINTING WITH DEFFERENT PERCENTAGE OF UNIFORM PIXELS MISSING
In general, missing is uniform, and each point determines whether it is lost or not according to a certain probability. We tested four synthetic images to restore images with different pixel uniform missing rates in the first experiment: Barbara image (256 × 256 pixels), Lena image (256 × 256 pixels), pixel uniform missing rates in the first experiment: Barbara image (256 × 256 pixels), Lena image (256 × 256 pixels), Cameraman image (256 × 256 pixels) and Mandrill image (256 × 256 pixels). Four original images are shown  in Figure 1. The reconstructed images are shown in Figure 2. Due to the outstanding performance of LDMM in image inpainting, we first considered the worst case. The observed image was obtained with the binary mask which had only 5% pixels, i.e. 95% pixels were randomly discarded. Then, we show the images inpainting results for 70% of the pixels missing, 50% of the pixels missing and 25% of the pixels missing respectively in Figure 2. In the experiments, we set the number of iterations to 100. Visually, we can easily observe that that the recovery result of our model is much clearer than those of BITV and KALS. Although KALS cannot restore large area pixel missing images, it has a strong ability to restore small area pixel missing images with obvious visual effects, such as the eyes of mandrill in Figure 2. BITV can restore images with the maximum pixel missing of 95%, which showed a better performance than what KALS can do, but our approach presented best results.
To reflect our model's ability to restored images with different proportions of missing pixels and quantify the performance of our approach, Table 1 shows comparison of our method with BITV and KALS methods by the PSNR, SSIM and time values. It can be revealed that our method provides the optimal PSNR, SSIM values in all examples, which are shown in bold face in Table 1. However, that although the SSIM value of KALS is higher than that of BITV in large area pixel missing, the visual effect is significantly lower than that of BITV. Therefore, we selected the image with 10% pixel missing in the subsequent experiment in order to make a better comparison. Regardless of the calculation time, our approach showed the best blind image inpainting capability, not only in the ability to repair large-scale missing pixel areas, but also in the high PSNR and SSIM values.
In Figure 3, we demonstrate blind image inpainting with 10% pixels missing. Twelve original images and their observed images with 10% pixels missing are shown in Figures 3(a) and 3(b), respectively. The observed images are applied to conduct the comparison. The inpainted results of our proposal, BITV and KALS are presented in Figures 3(c)-(f), respectively. It shown that our proposed method is the best performer among all three methods visually. The PSNR, SSIM and time comparison of twelve images are plotted in Figure 4. It can be seen that the PSNR and SSIM values of our method reach maximal values for different images. In view of the long execution time of our method in the previous experiments, we set the iterations to 50 in this experiment, but it can be seen from Figure 4(c) that the proposed approach still takes the longest time.

B. IMPULSE NOISE REMOVAL WITH DEFFERENT PERCENTAGE OF NON-UNIFORM PIXELS CORRUPTED
In the second experiment, we demonstrated that our method can also restore images with different non-uniform pixels corrupted rates. Here we take a non-uniform sampling depending on the polar coordinates of the pixels. The farther away they are from the center, the greater the probability of loss is. The four original synthetic images we used for the test are shown in Figure 5: Pepper image (256 × 256 pixels), Parrots image (256 × 256 pixels), Scenery image (256 × 256 pixels) and Villa image (256 × 256 pixels).
In the experiments, we set the number of iterations of our method to 100. The PSNR, SSIM and time values of the   inpainting results from all three methods are summarized in Table 2. The visual comparison of the inpainting results for 90% of the pixels corrupted and 10% of the pixels corrupted are shown in Figures 6 and 7, respectively. As can be seen from Figures 6 and 7, KALS achieves good visual inpainting effect in the structure and texture regions, especially in small area pixel corrupted images, which can be observed in the boundary of the Pepper, around the eyes of the Parrots, the reflection in the water in the Scenery, and the branches around the Villa. The inpainting effect of BITV in texture region  is obviously weaker than that of KALS. Our model is the best performer among all three methods, in particular, for ''Scenery'' with rich textures.

C. BLIND INPAINTING FOR RGB COLOR IMAGE
Currently, most of the images displayed in the medium are color images. Therefore, in this section, we present the experimental results to illustrate the inpainting ability of the proposed method for RGB color images in the case of uniform 200558 VOLUME 8, 2020 and non-uniform pixels missing. For the sake of brevity, we only show the visual optimization results obtained.
The images we used in the test are shown in Figure 8. The reconstructed images are shown in Figure 9. The visual results show through our method the image can be restored very well in both the cartoon part and the texture part, whether it is uniform or non-uniform pixels missing and corrupted, whether it is large-scale or small-scale pixels missing and corrupted.

V. CONCLUSION
In this paper, we proposed a low dimensional manifold approach for blind image inpainting and impulse noise recovery to simultaneously identify the image and the mask. Experiments were carried out on synthetic image. Moreover, quantitative measures were used to show the blind image inpainting ability of our model, and the blind image inpainting effect of our model is much better than those with existing methods.
As our model showed a long period of consumption in the calculation, we will continue to investigate the proposed method to obtain stronger blind image inpainting capabilities and less computation time than what the existing methods showed. Furthermore, our future work is to optimize the parameters of the model with the depth model after expanding the iterative formula of the model.