A guided edge-aware smoothing-sharpening filter based on patch interpolation model and generalized Gamma distribution

Smoothing and sharpening are two fundamental image processing operations. The latter is usually related to the former through the unsharp masking algorithm. In this paper, we develop a new type of filter which performs smoothing or sharpening via a tuning parameter. The development of the new filter is based on (1) a new Laplacian-based filter formulation which unifies the smoothing and sharpening operations, (2) a patch interpolation model similar to that used in the guided filter which provides edge-awareness capability, and (3) the generalized Gamma distribution which is used as the prior for parameter estimation. We have conducted detailed studies on the properties of two versions of the proposed filter (self-guidance and external guidance). We have also conducted experiments to demonstrate applications of the proposed filter. In the self-guidance case, we have developed adaptive smoothing and sharpening algorithms based on texture, depth and blurriness information extracted from an image. Applications include enhancing human face images, producing shallow depth of field effects, focus-based image enhancement, and seam carving. In the external guidance case, we have developed new algorithms for combining flash and no-flash images and for enhancing multi-spectral images using a panchromatic image.


Introduction
Smoothing and sharpening are two fundamental operations in image processing. Traditionally, smoothing is used to reduce noise, while sharpening is used to enhance details [1]. In recent years, smoothing has found increasingly more applications in graphics, computational photography, and computer vision. Edge-aware smoothing, which preserves sharp edges of objects, has been actively studied. Well known edge-aware filters include [2][3][4][5][6][7]. Among them, the guided filter [3] and its weighted versions [7][8][9] have the advantage of a low computational complexity of O(N ) in addition to their good performance. Applications of edge-aware filters include detail enhancement, flash On the other hand, sharpening is usually achieved through the unsharp masking algorithm [1]. Let I and S be the observed image to be processed and the sharpened image, respectively. The sharpened image is produced by S = I + γZ where γ is called the sharpening gain and Z is the output of a high-pass filter. The use of a linear filter to produce Z presents two main disadvantages in some applications: high sensitivity to noise and halo artifacts due to overshoot in high contrast regions. Non-linear filters were proposed in [10][11][12] to reduce the effect of noise but not solving the halo artifacts. Edge-aware filters are the main tools to combat the halo effect. In addition, there are many studies on using adaptive gain to perform content adaptive sharpening. For example, in [13], a pixel adaptive gain γ based on the dynamics of the image is proposed to sharpen areas of mid-range contrast, to avoid overshooting in high contrast regions, and to produce minimal sharpening at smooth regions. Attempts have also been made to formulate the adaptive sharpening problem as an optimization problem [14]. A similar approach was taken by [15] which used the local blurriness to vary the sharpening gain γ. The main goal is to avoid sharpening the very smooth background which is intentionally produced by the photographer to achieve the effect of shallow depth of field.
Although smoothing and sharpening are related through Z = I − J, where J is a smoothed version of I, and edgeaware filters are used to produce J to minimize the halo effect, these two operations are usually used in different applications. The main motivation of this work is to develop a unified framework such that smoothing and sharpening can be integrated in one filter whose function can be controlled by varying a parameter. Our aim is to develop a filter with a tuning parameter such that when it is set smaller/greater than 1 the filter is smoothing/sharpening. The filter must also have the edge-awareness property such that it does not blur edges when used in smoothing mode and does not create halos when used in sharpening mode. A distinctive advantage of such formulation is that it allows the user to perform selective smoothing and sharpening in different areas of an image to produce results such as smoothing the background while sharpening the main object. In addition, the unification of these two operations in one filter allows the user to have a better control in information fusion applications such as Pan-sharpening [16] and flash-no-flash imaging [3].
The main contributions of this work and organization of this paper are summarized in the following. Using the self-guided filter, we have developed adaptive smoothing-sharpening algorithms by extracting information of texture, depth, and blurriness to adjust the filter parameter to achieve content-aware processing. Applications include enhancement of images of human face, creating the effect of shallow depth of field, smoothing/sharpening guided by blurriness, and pre-processing an image to achieve better seam carving results. Using the filter in external guidance, we have applied the filter to combine images taken under flash and no-flash conditions, producing much better results than those produced by the guided filter. We have applied the filter to solve the Pan-sharpening [17] problem which combines information from multi-spectral images with a panchromatic image. Both subjective and objective comparison are discussed to validate the applications of the proposed filter.
2 The smoothing-sharpening filter, the self-guided filter, and main ideas of this work In this section, we first present a filter called the smoothing-sharpening filter in which smoothing and sharpening is configured through the setting of a parameter. Next, we revisit the basic idea of the guided filter. We then discuss the main idea of the proposed edge-aware smoothing-sharpening filter.

The smoothing-sharpening filter
We develop a unified framework for combining smoothing and sharpening into one filter. We first define the filter as J(n) = I(n) + (1 − α)∆I(n) (1) Figure 1: Magnitude response of the smoothing-sharpening filter. When 0 < α < 1, it is a low-pass filter. When α > 1, it is a high-frequency emphasis filter. When α < 0, the filter's function is not well defined.
where I(n) and J(n) are pixels of the input and output images at location n, α is a parameter, and ∆ is the discrete Laplace operator (which is referred to as the Laplacian operator in rest of this paper) defined as where µ(n) = 1 N m∈Ωn I(m) is the mean of the image calculated over a patch centered at location n. The symbol Ω n represents the set of pixel indices of the patch and N = |Ω n | is the number of pixels. The particular parameterization of this filter by 1 − α will become clear in the following discussions.
To demonstrate the characteristics of this filter, we consider a simple example of a 1-D filter of which the mean is calculated by an average filter µ(n) = 1 3 1 m=−1 I(n − m). The impulse response of the filter stated in (1) is then given by h(n) = {(1 − α), (1 + 2α), (1 − α)}/3. The frequency response is calculated as follows In Fig. 1, we plot the magnitude response (|H(ω)|) of this filter for three settings of α. We can see that (a) when 0 < α < 1, it is a low-pass filter, (b) when α > 1, it is a high-frequency emphasis filter, and (c) when α < 0, the filter's function is not well defined and is not considered in this paper.
Substitution of (2) into (1), we have which is a weighted average between I(n) and µ(n) when 0 ≤ α < 1. It is a low-pass filter. On the other hand, when α > 1 we can then re-write (4) as Equation (5) is the unsharp masking operation which is a sharpening filter. The equivalence of (5) and (1) shows that the filter stated in (1) can be configured as either a smoothing filter 0 < α < 1 or a sharpening filter α > 1. We can further re-write the filter stated in (4) in the following equivalent form which is the operation of a local mean minus a scaled local Laplacian of the signal.
To use this interpretation in the development of the edge-aware smoothing-sharpening filter, we need to generalize the concept of Laplacian in the following sense. In its original form, the Laplacian at a pixel location n is defined by (2) where the patch is centered at location n. The Laplacian can also be written as which is the average of the difference between the center pixel I(n) and each pixel I(m) in the patch. Using this interpretation, we make the following generalization. For any pixel I(q) at location q ∈ Ω n , the Laplacian is defined as This definition of the Laplacian thus generalizes the concept from the original one which only applies to the center pixel of the patch to the one which applies to all pixels in the patch. Although the original meaning of the Laplacian is lost in the generalization, we will use the same name in this paper to simplify the terminology. Using this generalization, the filter can be defined for all pixels in the patch as follows The difference between the two filters defined in (6) and (9) is that the former is defined for the center pixel of the patch while the latter is defined for all pixels in the patch. We will call the new filter (stated in (10)) a patch interpolation model, because the output is a weighted average of the input and the patch mean.
When the parameter α is fixed, the filter does not have an edge-awareness capability. This is a major problem of the filter. We will show in the next section that the idea of the guided filter provides a solution to this problem by adaptively setting the parameter α.

The self-guided filter
We revisit the basic idea of the guided filter [3]. A square patch of radius r has N = (2r + 1) 2 pixels. Let Ω k represent the kth patch in which a pixel at location q is denoted I k (q) where q ∈ Ω k and the subscript k indicates the patch. A linear model is imposed on each pixel of the patch such that where J k (q) is the desired output. The two patch dependent parameters a k and b k are determined by solving a regularized least squares problem with the following cost function where the second term is the regularization and is a user defined parameter. Solving ∂c/∂b k = 0, is the patch mean. By Substitution of this result into (11), we have another patch model which is an interpolation between the pixels in the patch and the patch mean.
A new regularized least squares problem is obtained by substitution of (13) into (12) which results in the following cost function where is the patch variance. Solving ∂d/∂a k = 0, we have Let I(p) represent the pixel to be processed at location p ∈ Ω k . It can be shown that the pixel I(p) belongs to N overlapping patches [3]. Since each patch model produces one output there are N modelling results {J k (p)} k=1:N . To aggregate these results, a weighted average [7,8] is performed: where N k=1 w k = 1. The original guided filter [3] uses the fixed weight w k = 1/N .

Main idea of the proposed filter
The key to our development is the patch interpolation model which is stated in (13) and the smooth-sharpening filter stated in (10). Comparing the two, we can see that they are in a similar form. There are two key differences.
• In its general form (1), the smoothing-sharpening filter can be configured as either smoothing or sharpening. It is not obvious how to set the parameter α such that the filter has the edge-aware capability.
• On the other hand, in the guided filter case, the filter parameter a k is specifically determined for edgeawareness. Referring to (15), for the case << σ 2 k which indicates strong texture/edge inside the patch, the algorithm assigns a k → 1 such that J k (q) → I(q). For the case >> σ 2 k which indicates a smooth patch, the algorithm assigns a k → 0 such that J k (q) → µ k . As a result, edge-aware smoothing is achieved. However, it is always a smoothing filter because a k = σ 2 k σ 2 k + < 1. Based on the above observations, to develop an edge-aware smoothing-sharpening filter we should find a generalization of the guided filter such that it is possible to set a k ≥ 1. We show in the next section that the development is based on two key ideas: 1. using the patch interpolation model stated in equation (13), and 2. using the principle of maximum a posteriori to determine the parameter a k to overcome the limitation of a k < 1 in the original guided filter.
3 The edge-aware smoothing-sharpening filter We first develop the self-guided edge-aware smoothing-sharpening filter in section 3.1. We then develop the externalguided version of the filter in section 3.2. Implementation and computational complexity of the proposed filter are discussed in section 3.3.

The self-guided smoothing-sharpening filter
We define the linear Gaussian observation model for the patch data as follows where r(q) is a realization of an i.i.d. zero mean Gaussian random variable with variance τ 2 N (τ > 0). Here we use the patch size N = |Ω k | to parameterize the noise variance such that the result does not depend on N .
To determine α k , we treat it as a random variable and use the principle of maximum a posteriori. More specifically, using Bayes rule we can write the negative log-posterior as a cost function D(α k ) by ignoring constants as follows where based on the observation model stated in (18), the negative log-likelihood for the patch is given by Compared with the first term of the cost function of the guided filter stated in (12), the above negative log-likelihood has an extra parameter τ . We set τ = 1 in this work which allows us to include the guided filter as a special case in the proposed filter.
Substitution of (20) into (19) the cost function can be written as Compared with the cost function of the guided filter, the above cost function is different in two aspects: (a) the parameter b k is implicitly defined in the patch interpolation model, and (b) the regularization term 2 α 2 k (which is the negative of logarithm of zero mean Gaussian) is replaced by the negative log-prior which permits us to develop different filters.
In this work, we consider the generalized Gamma distribution as the prior: where θ > 0 is a scale parameter. We set h = 2 and η ≥ 0 to control the shape of the distribution. Substituting (22) into (21) we have the cost function in which constant terms are omitted We can easily see the motivation and justification of such settings. When η = 0, the cost function is the same as that of the guided filter with the setting = 1/θ 2 . When η > 0, the cost function has an extra term −η log α k compared with the cost function of the guided filter. We will show that this extra term permits the filter to be configured as either a smoothing filter (0 < α k ≤ 1) or a sharpening filter (α k > 1).
Another justification is mathematical simplicity. The cost function under this parameter setting is convex leading to a unique minimum. Indeed, the generalized Gamma distribution allows us to explore other settings of parameters such as h = 2. However, for such a setting, the cost function may not be convex and is difficult to optimize. Therefore, we do not pursue study in this direction.
We now determine the filter parameter α k by minimizing the cost function D(α k ) which is equivalent to maximising the posterior. Solving ∂D/∂α k = 0, we obtain the optimal value

Properties
To reveal how the proposed filter generalizes the original guided filter, we set = 1/θ 2 such that the guided filter's parameter stated in (15) is given by Substitution (25) into (24), we have We can clearly see that the parameter of the proposed filter can be expressed as a function of the parameter of the original guided filter. Their relationship is discussed in the following. Based on the interpolation model stated in equation (13), we can prove that the proposed filter can be configured by setting η relative to as follows.
• When η < , we can show that α k < 1 which leads to a smoothing filter. In an extreme case when η = 0, the proposed filter is reduced to the original guided filter. It is also interesting to note that for this setting α k > a k which means the proposed filter always performs a lower degree of smoothing than the original guided filter.
• When η = , we can show that α k = 1 which leads to no filtering.
• When η > , we can show that α k > 1 which leads to a sharpening filtering. The sharpening gain is defined as γ k = α k − 1 which is an increasing function of η. Figure 2: Interpolation weight α k as a function of patch variance σ 2 k and filter parameter κ under two fixed settings of (top = 0.001, bottom = 0.01). As κ is increased, α k is less adapted to σ 2 k and is close to the constant 1 for In light of the above discussion and to simplify parameter settings, we introduce another parameter κ to replace η by defining η = κ/θ 2 = κ . The parameter of the proposed filter can be re-written as The advantage of this new parameterization is that the filter can be configured as smoothing by setting 0 ≤ κ < 1 and as sharpening by setting κ > 1.
For the smoothing case, we plot α k for different settings of κ including the case with κ = 0 which is the guided filter. Results are shown in Fig. 2. We can see that as κ is increased, α k is less adapted to σ 2 k and is closer to the constant 1 (no filtering) for σ 2 k > T where T is a signal dependent threshold. Next, we study the sharpening gain γ k as a function of κ (κ > 1) and the patch variance σ 2 k . The interpolation model can be re-written in a sharpening filter form Fig. 3 shows the sharpening gain as a function of the patch variance σ 2 k for various settings of κ. We can make the following observations. (a) The sharpening gain is a decreasing function of the patch variance. This is a desirable property of the sharpening filter. It performs a higher degree of enhancement on a patch of smaller variance which is usually due to low contrast. (b) The level of enhancement can be controlled by setting the parameter κ. A bigger value will lead to a bigger sharpening gain for the patch of fixed variance. (c) The parameter also controls the sharpening gain. For the same setting of κ, a bigger value of will lead to a bigger value of the sharpening gain.

The guided form and its properties
In this section, we develop a guided version of the proposed filter by a further generalization of smoothing-sharpening formulation stated in section (2.1) and the guided filter. Figure 3: The sharpening gain γ k is a decreasing function of the patch variance σ 2 k for various settings of κ under two fixed settings of (top = 0.001, bottom = 0.01). The sharpening gain γ k is bigger for a bigger value of κ and is a decreasing function of the patch variance σ 2 k .

The guided smoothing-sharpening formulation and the guided filter
Referring to the filter formulation stated in (9), we can see that one of the key components is the Laplacian which is a second derivative operation and is thus sensitive to noise in the image I. When a guidance image G, which is assumed to have a higher signal-to-noise ratio, is available, a generalization is to replace the Laplacian calculated on I by the Laplacian calculated on G such that the filter can be written as: where µ(n) is the mean of the patch centered at I(n) and Although this idea is technically sound, how to determine α remains a problem. We will revisit the basic idea of the original guided filter and show how this problem can be solved by using a similar approach as the development of the self-guided version of the proposed filter described in the previous section.
In the original guided filter, the patch model is given by The two parameters a k and b k are determined by minimization of the cost function over the patch data It can be shown that where µ k and ν k are the mean for the kth patch of image I and G, respectively. The parameter a k is given by where ς 2 k is the patch variance of G, and φ k is the sample covariance Substituting (33) into (31), we have a new patch model where ∆G k (q) = ν k − G k (q) is the generalization of the Laplacian defined in (8).
Comparing the guided filter model stated in (36) with the proposed smoothing-sharpening model stated in (29), we can see that they are similar. Simply put, if we start with the filter model stated in (29), we will determine the same filter as the guided filter which is a smoothing filter. In the next section, we address the problem of how to develop a smoothing-sharpening guided filter.

The proposed guided smoothing-sharpening filter
To use (36) to develop the smoothing and sharpening filter, we face a new difficulty that a k can be positive or negative. This is unlike the case for the self-guidance version developed in section 3.1.1 where it is always the case a k ≥ 0.
Referring to (34), the sign of a k is defined by the sign of the covariance φ k such that The role of the sign of the covariance φ k can be explained as follows. Because ∆I k (q) is replaced by ∆G k (q), it requires that the two patches I k and G k must be correlated. If they are positively correlated, then it is expected the two Laplacians are of the same sign. If they are negatively correlated, then it is expected the two Laplacians are of opposite sign. Thus a correction of the sign of the Laplacian calculated on the guidance image is required. The definition of a k stated in (34) automatically satisfies this requirement.
In light of the above discussion, we can develop the guided version of the smoothing-sharpening filter by defining the guided patch interpolation model as the following such that α k is a positive parameter. We can then follow the same procedure as that presented in section 3.1.1 for the development. More specifically, the negative log-likelihood is where we have set τ = 1 as before. Using the same generalized Gamma distribution as the prior, we obtain the negative log-posterior as the cost function D(α k ) We also follow the same parameter settings of the self-guided form by letting = 1/θ 2 and η = κ . Solving ∂D/∂α k = 0 and re-arranging the results, we have

Properties
We can clearly see that when G = I, we have ς 2 k = φ k = σ 2 k . The guided version of the proposed filter stated by (42) reduces to its self-guided version stated by (26). The relationship between the proposed filter and original guided filter can be revealed by substitution of (34) into (41), which results in the following We can also see that α k ≥ |a k | from (42). The original guided filter is a special case of the proposed filter when κ = 0 leading to α k = |a k |.
Next we discuss under what parameter setting the filter is smoothing or sharpening. The analysis presented in section 2.1 can not be used, because the Laplacian is calculated on the guidance image rather than on the image to be processed. The following analysis is based on an observation that a smoothing filter will reduce the patch variance while a sharpening filter will increase the patch variance. It is also assumed that the patch mean is not changed by the filter. This is a reasonable assumption because smoothing and sharpening usually do not change the average brightness of the image.
To make the discussion easy to follow, we first define the patch variance for the three images as Guided image : In equation (45), we substitute J k (q) by the patch interpolation model stated in (38) and use the definition stated in (44), we obtain Next we identify parameter settings that lead to reduced variance, i.e., τ 2 k /σ 2 k ≤ 1 for smoothing or increased variance, i.e., τ 2 k /σ 2 k > 1 for sharpening. It can be shown that k + and ρ k (|ρ k | ≤ 1) is the cross correlation coefficient of the two patches defined as where φ k is the sample covariance between the corresponding patches of the original and guided images. It is defined in equation (35).
In a special case in which κ = 0 leading to the original guided filter, we have the following results This means that the original guided filter is always a smoothing filter.
However, there is a highly non-linear relationship between the ratio τ 2 k /σ 2 k and the two filter parameters κ and for the two patches in I and G where the variances (σ 2 k and ς 2 k ) and correlation coefficients (ρ k ) can be calculated. In theory, we can calculate the required parameter settings such that for two given patches the ratio is greater than one or less than one. But doing so adds considerable computation burden in practice. A practical approach is to let the user set the two parameters such that the desirable result is produced. In this regard, we can see that the ratio τ 2 k /σ 2 k is an increasing function of κ σ 2 k . For the case where and σ 2 k are fixed, increasing κ will increase the ratio and the patch is more likely to be sharpened (or less smoothed). For the case where κ and are fixed, a patch with a larger/smaller value of variance is less/more likely to be sharpened. This is a desirable feature because more sharpening should be applied to areas of less variance.

Implementation and computational complexity
The implementation is similar to that of the original guided filter. For each patch, we calculate the parameter α k using either (27) for the self-guided version or (41) for the guided version. We then perform the weighted average operation. More specifically, for the self-guided version, the filter output is given by For the guided version, the filter output is given by where β k = sign(φ k )α k and a k is given by (34).
We calculate the weight for the self-guided and the guided case as follows and where c k is a normalization factor to ensure N k=1 w k = 1, s is a user defined scale parameter,σ 2 andς 2 are the average of σ 2 k and ς 2 k over the whole image. The proposed guided smoothing-sharpening filter can be implemented in MATLAB of which the code is shown in Appendix. The self-guided version has a similar implementation. We assume the four parameters are given: patch radius (r), Kappa (κ), Epsilon ( ), and Scale (s). We can see from the brute-force implementation that the proposed filter has an O(N ) complexity which is the same as that of the original guided filter. It can be implemented by using 7 linear filters.
For color images, we can process each color component individually. Alternatively, we can convert the image from RGB to HSV color space. Filtering is performed on the value channel. The processing result is then combined with the hue and saturation channels and is converted back to RGB.

Applications examples
There are two purposes of this section: validation (section 4.1) of the theoretical analysis of the proposed filter, and demonstration of successful applications in (a) adaptive smoothing and sharpening (section 4.2) based on extracted information of texture, depth and blurriness, and (b) information fusion (section 4.3) for denoising and creating high resolution multi-spectral images.

Effects of parameter settings for the self-guidance case
We demonstrate the properties of the filter and confirm the theoretical analysis presented in section 3.1.2. The proposed filter has 3 user defined parameters: (a) The patch radius r, (b) the sharpening/smoothing gain κ, and (c) the regularization parameter . In this section we study the effect of these parameters on the processed image.
In Fig. 4 we demonstrate the effect of varying κ by keeping the rest of the parameters fixed. Fig. 4a and 4b show the effect of smoothing when 0 < κ < 1. Smaller κ values increase the smoothing level on the result image. Fig. 4d and 4e show the effect of sharpening when κ > 1. Larger κ values produce a sharper result. The image shown in Fig. 4c is produced by setting κ = 1. We can verify that it is exactly as the original image. Thus when κ = 1 the filter produces no smoothing or sharpening effect.
Next, we study the effect of patch size and for sharpening and smoothing separately. To set the filter in smoothing mode we set a fixed κ = 10 −2 . In Fig. 5 the results are organized in such a way that the radius varies from 5 to 10 column-wise while varies from 10 −2 to 1 row-wise. Results shown in this figure clearly show the edge preserving capabilities of the filter in smooth mode. Setting a larger value produces a more washed out result, while increasing the radius of the filter also produces a stronger smoothing result.
The results shown in Fig. 6 are obtained by setting κ = 20 to demonstrate the sharpening effects as a function of different and r values. These results confirm that the sharpening gain increases with the increase in as previously stated in Fig. 3. Increasing the radius impacts the variance for each pixel position. As a result, it produces a change in the sharpening gain.

Application in adaptive smoothing-sharpening
As mentioned in the previous section, the value of κ controls the sharpening/smoothing gain of the proposed filter, in this section we propose three pixel-adaptive smoothing and sharpening algorithms by defining κ as a non-linear transformation of a feature map such as depth, blurriness or texture.
The non-linear transformation used in this paper is simply a variation of the Gompertz 2 function which is a sigmoid function. It is defined as: where a, b and c are three parameters. Let us consider t (t ∈ [0, 1]) as a feature map extracted from the input image, the parameter of the proposed filter κ is defined as: where κ min and κ max are the minimum and the maximum values that the gain κ will take, c is the growth rate of the transformation and t 0 is the value of t that produces κ i = (κ max − κ min )/2.
We show an example of the non-linear transformation in Fig. 7, where there are two well defined areas in the function: the smoothing region where the κ min ≤ κ < 1 and the sharpening region where 1 < κ ≤ κ max . We can control the level of smoothing or sharpening by changing the values of κ min and κ max . Setting κ min = 1 will cancel the smoothing effect also setting κ max = 1 will cancel or not produce any sharpening on the image. In the following subsections we produce content adaptive κ by using the non-linear transformation on feature maps.

Texture guided smoothing and sharpening of face images
A challenge in sharpening portraits by a non-adaptive unsharp masking algorithm is that undesirable effect on skin regions is usually produced. Examples are shown in Figures 8(b) and 9(g,h) in which the skin part of the image is sharpened. To solve this problem, we first estimate a binary skin map using one of the many algorithms for skin segmentation, e.g., [18][19][20]. We then transform the binary skin map by using the non-linear transformation in (55) to obtain pixel-adaptive κ, which is used in the proposed filter to sharpen non-skin regions only while gently smoothing the skin region to produce a notable face enhancement. In Fig. 8 we compare the result of the proposed method against the popular unsharp masking (UM). UM algorithm has a fixed gain (γ) to amplify the high frequency components of the image. It can be seen that our method produces a more aesthetically pleasing result in skin region while still sharpening the non-skin region.
In Fig. 9 we compare our method with other state-of-the-art sharpening methods such as generalized unsharp masking (GUM) [21], unsharp masking [1] and contrast adaptive sharpening (CAS) 3 . Our filter produces a more natural and aesthetically appealing effect on the image than UM and its performance is similar to GUM and CAS.

Depth guided smoothing for shallow depth of field
Modern mobile phones can have multiple high resolution cameras to capture high quality images which can be used to estimate the depth map of a scene. We combine this capability and the proposed algorithm to produce a shallow depth of field (SDoF) effect which is frequently used to emphasize the main object. Traditionally, SDoF effect is achieved by using a lens with a large aperture in a SLR camera. The proposed algorithm of pixel-adaptive smoothing and sharpening can be used as post-processing tool to create the SDoF effect. The idea is to obtain the depth map from the phone and perform the non-linear transformation to determine κ which is used in the filtering process. In this experiment, we used a Samsung Galaxy Note 9 phone in live focus mode to capture the image and depth map information. We note that methods such as [22,23] can be used to estimate the depth map from a single image if only one camera is available.
The depth map D is a gray-scale image with values in the range [0,1]. A closer object in the scene has a smaller D value. So the feature map is defined as t = 1 − D. Applying the non-linear transformation, the depth information is mapped to κ which is a decreasing function of D leading to progressively smoothing effect as the distance increases. At the same time the closer objects are sharpened to correct slight out of focus or blur. In Fig. 10 and Fig. 11 we show two different results where the foreground is sharpened and the background is smoothed to produce the SDoF effect. We can see that the background is smoothed with a natural appearance to simulate the defocus blur, also the foreground is sharpened reducing the blur due to movement in Fig. 10 and out of focus in Fig. 11.

Content-aware seam carving
We present an application of the background smoothing algorithm detailed in previous section as a pre-processing step for seam carving. Seam carving was introduced in [24] as an effective tool for resizing an image without significant change to main objects. The idea is to delete pixels of unimportant details in an image. The importance of a pixel is measured by a function of gradient. A natural image often contains details such as trees, sand, grass which are usually less important compared to the object of interest such as human. However, a direct application of gradient-based seam-carving may lead to unsatisfactory results. An example is shown in Fig. 12b. A solution to this problem is to use content-aware image resizing. To avoid elimination of information in foreground, we pre-process the input image by using the proposed SDoF algorithm which not only smooths out details in the background but also emphasizes the

Blurriness guided sharpening and smoothing
When an object in a scene is outside the focal plane it is defocused with a blur level directly proportional to the distance from the focal plane [25]. Due to the limited depth of field intrinsic in most optical systems, the defocus blur is present in most of images. When the depth information is not available, the defocus blur becomes the simplest depth cue in an image [26] and it is widely used by photographers to make the main object of the scene to stand out from the background. Sometimes, due to wrong focal settings, images need to be sharpened or deblured to achieve a more pleasant result.
When an image has defocus blur present, the sharpening process becomes a challenge. Sharpening highly defocused regions produces artifacts and sharpening in focus regions can lead to over-sharpening. To tackle these problems we propose a method by sharpening and smoothing an image adaptively using the defocus-blur map to compute a spatially varying κ map. The key behind the success of our method is the estimation of the defocus-blur map.
Methods have been developed to estimate the defocus-blur map. Some of the methods use multiple images or special hardware [27][28][29] while others use a single image. These methods fall into two main categories. The first one is the traditional image processing methods such as [30,31] which perform a frequency domain analysis to estimate the defocus-blur map or [32,33] which use changes in gradient to estimate the blur level at edges and then interpolate those level to the rest of the image using a matting algorithm. The second one is the machine learning based methods such as [34] which uses an end-to-end CNN to estimate the defocus map and [35] which estimates the blur map by using a regression tree field (RTF) model.
In this work we use entropy, which is a measure of pixel variation in a local area, as an indicator for the defocus. The defocus-blur map D at a pixel location is defined as the entropy of a patch centered at that particular pixel. The map is then refined by using the guided filter [3] which uses the original image as the guidance. The refined defocus-blur map is non-linearly transformed by equation (54) to obtain the desired κ map. The proposed filter with the pixel-adaptive κ is applied to the image. The result is adaptive smoothing-sharpening based on the local entropy information as a measure of defocus-blur. In Fig. 13 we first show an image which contains different levels of focal blur. The boy's face (image (a)) is in focus while the background is slightly out of focus. The refined entropy map (image (b)) is produced by the MATLAB function entropyfilt with a window size of 33x33 pixels and the result is refined by using guided filter [3] (r = 32, = 0.01). The κ map is shown in image (c).
In the second row of Fig. 13 we show 3 different results. Image (d) is the result of smoothing areas with some degrees of focal blur. The background is successfully blurred while the boy's face remains sharp leading to a pleasing SDoF effect. Image (e) is the result of sharpening the out-of-focus regions to increase the depth of field. The background appears sharper producing the sensation that the scene's depth is slightly larger than the original. In both cases, the boy's face which is in focus, is not changed (achieved by setting κ = 1). However, we should point out that the proposed filter can be easily configured to sharpen the in-focus objects by adjusting the parameters of the non-linear transformation. As a comparison, image (f) shows the effect of sharpening with a fixed κ value for the whole image, leading to an image of non natural appearance.
To further demonstrate the advantage of the proposed smoothing-sharpening filter, we compare results from the gradient domain guided filter (GDGF) [9], the weighted guided filter (WGF) [8], and the side window guided filter (SWGF 4 ) [36]. We tune parameters of these 3 filters such that the background is maximally smoothed while preserving information of the face shown in Fig. 13(a) as much as possible. Results are shown in Fig. 14 which clearly shows that the proposed filter has the best capability for blurring the background while preserving the information of the face which is almost unchanged. On the other hand, both GDGF and WGF can blur the background to some extend at the cost of smoothing the face. For the SWGF, we have to choose a relatively small patch size to prevent the face being overly smoothed. As a result, there is little effect on smoothing the background.
(a) Proposed method (b) GDGF [9] (c) WGF [8] (d) SWGF [36]  In this section we demonstrate another application of the proposed filter where guided smoothing and sharpening is required. For example, a picture taken under low light condition contains a high level of noise due to the use of high ISO setting. One of the successful applications of the original guided filter [3] is in processing images captured under low light condition without using the flash. One such image is shown in Fig. 15a. The idea is to use another image captured with flash-on as a guidance to enhance the one without flash. In the original implementation, the guided filter with parameters r = 8 and = 0.004 are used. Result is shown in Fig. 15(c), where we can see that noise has been greatly reduced while the color information is preserved. This is however at the cost of loss of details, e.g., details on the wall and on the vases. This is evident when we compare the result with the guidance image (with-flash).
To tackle this problem, we first apply the guided filter in an iterative manner. We represent the filter operation as J = GF (I, G) where GF denotes the guided filter. The iteration is performed as: J (0) = I and J (n) = GF (J (n−1) , G).
Using the parameter settings r = 25 and = 10 −6 , we perform 10 iterations and show the result in Fig. 15(d).
Comparing the original GF result with the iterative GF result, we can see that the latter has retained more details of the original scene than the former.
Next, we test the proposed guided smoothing-sharpening filter in the same iterative way, i.e., same parameter settings with 10 iterations. The proposed filter has two extra parameters: κ and the scale s. For simplicity, we set s = 1 and κ = 10 and 100 to study the sharpening effect. Results in Fig. 15(e) and (f) which show that the proposed filter does indeed produce sharper results than the iterative GF. To make a quantitative comparison, we calculate the total variation of the image. The total variation for image I is defined as where I (c) is the first derivative of the image along the horizontal/vertical direction at location n and the superscript c is used to indicate the color channel. Since the total variation is the sum of absolute values of the first Table 1: The total variation of the image produced by the proposed iterative smoothing-sharpening filter. It is an increasing function of κ indicating the image appear to be sharper by using a larger setting of κ. derivative, we can use it as an indication of the sharpness of the image in this application. The total variations for different settings of κ are shown in Table 1, where κ = 0 corresponds to the iterative guided filter. We can see that the sharpness of the image is indeed an increasing function of κ.
Another issue is related to the number of iterations. In general, for κ > 1, more iterations tend to produce a higher degree of sharpening effect. How to set the number of iterations is application dependent and it can be a user specified parameter. For the image shown in Fig. 15, we empirically found that between 5 to 10 iterations and a setting of 10 < κ < 100 result in visually pleasing images.

Pan-sharpening
Multi spectral (MS) images usually have low spatial resolution but are rich in spectral information. On the other hand panchromatic images (called PAN images) have low spectral resolution but have high spatial resolution. For example, the IKONOS and QuickBird imaging sensors capture a PAN image with a spatial resolution of 1 and 0.6 m respectively and a MS image with a spatial resolution of 4 and 2.6 m respectively [37]. Pan-sharpening is a technique that combines information of MS images with PAN images to produce a high spatial resolution image with large spectral information. Pan-sharpening is a useful tool in many remote sensing applications.  Pan-sharpening techniques have been continuously developed over the years. Some frequently used techniques include: Brovery transform (BT) [37], intensity-hue-saturation (IHS) [38], principal component analysis (PCA) [16], wavelet transform (WT) [39,41], guided filter based methods such as [17,42], and P+XS [40]. We use some of these methods to compare with the result of the proposed algorithm.
The proposed method consists of 3 steps similar to the GF-based approach [17]. The MS image is first up-sampled using nearest neighbor interpolation. The result is then processed by the proposed filter which uses the PAN image as a guidance to transfer the high spatial resolution information from the guidance image to the low resolution MS image. The last step is a histogram matching on the result image using the original MS image as a reference. Fig. 16a and Fig. 16b show the up-sampled MS image and the PAN images respectively. The images are from United States Geological Survey database 5 . Fig. 16c shows the output of the proposed algorithm using r = 11, = 0.1, κ = 1.2, scale = 0.5, the details of the PAN image are successfully transferred to the up-sampled MS image and the spectral resolution is preserved. In the second and third rows of Fig. 16 we present the results of the Pan-sharpening process using BT, IHS, PCA, WT, GF and P+XS algorithms. We can observe that our method produces sharp and high contrast results with large spatial and spectral information.
We use ERGAS [43] to quantify the quality of a Pan-sharpened image. ERGAS is a metric that calculates the spectral distortion. Ideally, its value should be zero. The ERGAS values for results produced by different methods are shown in the caption of Fig. 16. We can see that the result of the proposed filter is of about the same quality as those produced by state-of-the-art methods. Thus, the proposed filter is a new tool for Pan-sharpening with the ability to perform pixel-wise sharpening or smoothing.

Conclusions
Smoothing and sharpening are two fundamental operations in image processing. They are usually related through the unsharp masking algorithm. In this paper, we have developed a new filter which can perform smoothing and sharpening depending on the setting of a parameter κ. The filter is a smoothing filter or a sharpening filter when 0 < κ < 1 or κ > 1. The systematic unification of these two operations in one filter is based on (a) a new Laplacian based filter formulation which unifies the smoothing and sharpening operations, (b) a patch interpolation model similar to the guided filter which provides the edge-awareness capability, and (c) the generalized Gamma distribution as the prior for parameter estimation. As a result the filter allows pixel-adaptive image smoothing/sharpening by adapting κ to local characteristics such as texture, depth, and blurriness. Based on the patch interpolation model, the proposed filter uses the guidance information in two ways. In self-guidance the proposed filter uses information of the image to be processed and has the ability to use other information to adapt κ. In external-guidance, the filter is similar to the guided filter, but has an extra ability of adaptive smoothing-sharpening. In addition, the proposed filter has the desirable edge-awareness property which retains sharp edges in smoothing and does not suffer from the halo effect in sharpening.
Using the filter in self-guidance we have developed adaptive smoothing-sharpening algorithms based on information of texture, depth and blurriness to enhance human face images, to create the effect of shallow depth of field, to perform adaptive processing based on local blurriness, and to pre-process an image to achieve better seam carving results. Using the filter in external guidance, we have combined images of under flash and no-flash conditions, producing much better results than those produced by using the guided filter. We have also demonstrated the successful application of the filter to solve the Pan-sharpening problem which combines information from multi-spectral images with a panchromatic image.