An Effective Algorithm for Specular Reflection Image Enhancement

Specular highlights often exist in real-world images due to the material property of objects and the capturing environments. High light in real-life scenes can reduce image quality and cause loss of image information, to overcome this, we propose an effective specular reflection image enhancement algorithm. First, the scene depth map of specular reflection image was obtained by the color attenuation prior method; then, two adaptive adjustment factors were developed to optimize the transmission map for specular reflection images and L0 gradient minimization filter was used to eliminate halo artifacts. Finally, the nonuniform illumination compensation strategy based on the YUV color space was used to expand the dynamic range of the image and increase the saturation of the image. Experimental results via subjective and objective evaluations demonstrate that the proposed method can provide better visual quality than other methods and can effectively enhance the information in the specular reflection image.


I. INTRODUCTION
Digital images captured under discrete source illumination often contain specular reflections, which conceal object surface color and useful texture features. The saturated highlight will even lead to the complete loss of the information of the local area of the object. The presence of highlights in an image not only affects the quality of the image, but also causes great interference to the computer vision applications such as segmentation [1], object detection or recognition and scene reconstruction [2][3], so it is very important to enhance the region covered by highlight.
dichromatic reflection model-based approaches. The dichromatic reflection model represents the image as a linear superposition of the specular reflection component and the diffuse reflection component. This model has been widely used for specular highlight removal. Shen et al. assumed that there were only two types of pixels in an image, normal and highlighted, and then used chroma to calculate the reflection components of the two types of pixels [4]. Yang et al. used specular-free images to cluster pixels and then restored the diffuse color in each cluster [5]. He et al. separated specular reflection based on matrix decomposition technology [6]. Yang et al. estimated the maximum diffuse chromaticity values of specular pixels by directly applying a fast bilateral filter [7][8]. Ren et al. introduced a color line constraint into the dichromatic reflection model and proposed a fast specular removal method [9]. Jie et al. assumed that the highlighted region of the image is sparse and separated the highlight by solving the sparse matrix [10]. Suo et al. [11] extended the dichromatic reflection model in terms of L2-normalization and by formulating the problem such that the illuminant is orthogonal to the chromaticity. Their approach also requires clustering for estimation of region-specific purely diffuse colors. The technique [12] proposed a tensor voting method and a sparse non-negative matrix factorization (NMF) method to separate the high-light components. Li et al. [13][14] presented a method for removing specular high-light in facial images. Son et al. [15] proposed a convex optimization framework to effectively remove the specular highlight from chromatic and achromatic regions of natural images. Xia et al. [16] proposed a global optimization method for specular highlight removal. The method consists of estimating diffuse chromaticity by correcting hue and saturation on highlighted regions and estimating diffuse and specular reflection coefficients using convex optimization with double regularization. Fu et al. [17] present a novel specular highlight removal method on the observation of real-word images. Ramos et al. [18] proposed a highlight removal method based on dichromatic reflection model and operates through histogram matching in the YCbCr color space. These dichromatic reflection model-based methods have limited performance with images with text, because they often wrongly detect white text texture as highlight. Early methods often perform a color segmentation which is not robust to complex textures, dark pixels are sometimes introduced on the highlight region in the recovered image.
image-processing approaches. Nayar et al. proposed a method that combines color information with polarization to obtain color information of multiple images from different polarization angles to weaken the highlight region [19][20]. Sato and Ikeuchi analyzed the color features of multiple images taken by moving light sources and used the dichromatic reflection model to separate the specular reflection component [21]. By assuming that surface geometry is known, Wei et al. [22] leveraged the principal component analysis to separate highlights and estimate the position of light source. Guo et al. [23] proposed RPCA, which focuses on removing specular reflection from superimposed multiple images. Because methods these methods use several images, they are not suitable for general situations. In recent years, the image formation model has also been used for highlight removal. For most natural images, the dark channel provides an approximate specular free image [24]. Xin et al. obtained specular-free images through improved dark channel priors and enhanced the information affected by the specular region [25]. The image recovered by the dark channel model will be darker, and the image quality will have degraded. There are still some problems with low contrast, color distortion, and under-/overexposure after image processing. Recently, some methods aim to transform low dynamic images containing low illumination and high illumination into high dynamic images with rich colors and much brightness levels. Sahar et al. obtained an HDR-like image from a single LDR image by removing the specular reflection component of the highlight pixel and the enhanced image by combining low-illumination enhancement technology and specular removal technology [26][27]. Image-fusion techniques were also used to generate high dynamic image from low dynamic images, they can provide reliable and comprehensive image information from source images [28]. By generating multi-level illuminations from a single image, Zhu et al. [29] obtain high-visibility images with enhanced contrast and proper sharpness of scene objects. When the reflection is strong, their contribution to the brightest area is very little.
Although current specular highlight removal algorithms have obtained some achievements, there are still following problems. First, the current research only focuses on removing the highlight component from the image, and does not involve recovering the information of the highlight image. Second, most of current algorithms remove specular highlight for medical images and specific-images, for specular highlight images in real scenes, the existing algorithms cannot remove the highlight components in the image very well.
Based on the drawbacks of current algorithms that deal with specular reflection images, this paper proposes a new method that can improve the contrast and chromaticity of highlights-degraded image. Compare with the previous methods, the method proposed in this paper offers the following contributions: 1) Our research focuses on removing specular components from specular reflection images in real scenes and enhancing the information affected by the highlighted regions. Compared with the existing methods, our method can enhance the texture information of the area affected by highlight, so that the image information is clear and easy to identify. We also assume that the reflection is not fully saturated, because when the reflection is saturated, all three channels of any pixel have the highest value, it is very unlikely to achieve any information from that pixel even if in the real scene there is information like color and texture. 2) This method is mainly aimed at specular reflection images in real scenes, which have certain research significance and practical application value. Our method can reveal the details in both high illumination and low illumination areas of an image, especially the low-light images with a reflection object. The experimental results shows that our method can be used for a wide range of images. The remainder of this paper is organized as follows. Section II will give the proposed algorithm framework and introduce the method in detail. Section III will introduce the experimental data of the different scenes, show the experimental results, and verify the feasibility of the method. Finally, the conclusions are drawn in Section IV.

II. PROPOSED METHOD
In this section, we propose an effective specular highlight enhance algorithm based the image formation model. Fig. 1 illustrates the framework for our proposed algorithm. First, VOLUME XX, 2017 9 we quickly obtain the scene depth of the image by the color attenuation prior method, and then estimate the transmission map using the depth map. The transmission map calculated from the estimated depth map is called the raw transmission map, and is usually not good enough for highlight removal. According to the characteristics of the specular reflection image, we propose two adaptive adjustment factors to constrain the transmission map. To avoid the halo and pixel-clustering artifacts of specular reflection images, an L0 gradient minimization filter is used to refine the transmission map. The images obtained by the color attenuation prior technique suffer from dullness and higher illumination variations due to the consideration of a constant value of atmospheric light. We removed such variations by compensating for nonuniform illumination based on the YUV color space

A. COLOR ATTENUATION PRIOR
The image formation model is widely used in computer vision and image processing [30]. The model is described as where (x,y) is the image coordinate, I is the original image, and J(x,y) is the restored image. β is the medium scattering coefficient, which indicates the ability of the atmosphere to scatter light per unit volume. d is the scene depth of the scenic spot in the image field at x, A is the global atmospheric light, and t is the transmission map, which is inversely proportional to the scene depth.
Color attenuation prior is the linear model of the correlation between scene depth, image saturation and brightness difference [31]. The model is described as where v and t are the brightness value and saturation of the image, respectively, and (θ0, θ1, θ2) are the constant parameters of the model. ε is a random variable representing the model error. A simple and effective supervised learning method is used to learn the coefficients (θ0, θ1, θ2). The optimal learning results were θ0 = 0.121779, θ1 = 0.959710, and θ2 = -0.780245. Because white objects usually have very low saturation and high brightness, the depth calculated using Equation (3) is filtered by a minimum filter to reduce the influence of white objects on the accuracy of the estimation process. However, due to patch-wise local minimum operations, this depth map generates halo artifacts. Therefore, a guided filter was applied to remove halo artifacts (GF) [32]. The method for estimating atmospheric light is similar to that of dark channel prior, the top 0.1 percent brightest pixels in the depth map are chosen and, among these pixels, the one with the largest L2 norm in the input image is selected as the atmospheric light [33]. The highlight regions of images usually have high luminance, but low contrast and color saturation, similar to hazy images [25], therefore scene depth of the specular reflection image can be obtained by color attenuation prior. After estimating the scene depth map d and the global atmospheric light A, the raw transmission map t is estimated according to Equation (2).  We use the color attenuation prior to quickly estimate the raw transmission map of the specular reflection image. Different from the degradation mechanism of fog images, specular reflection images do not have a uniform degradation model. Because they containing large highlight areas, the depth of the high brightness area in specular reflection images will change violently [25]. The attenuation prior uses a fixed value to restore the image pixel values, which is not suitable for highlight removal. We didn't simply constrain the transmission map between two fixed boundaries, 0.05 and 1, because the small-global limit (0.05) indicates that highlight component of regions with this lower bound will be heavily removed, possibly resulting in color distortion. We define two adaptive adjustment factors from the characteristics of the specular reflection images to refine the transmission map.

B. ADAPTIVE CONSTRAINTS
First, the transmission map can be derived from Equation When the specular-free image is recovered, its normalized value will lie within the range 0 to 1. Since J ≥ 0, the transmission map can be obtained by combining Equation (4): We can get the first adaptive constraint as The second adaptive constraint is inspired by the meaning of the transmission map; the smaller the transmission values are, the more highlighted components are removed. The transmission map calculated from the estimated depth map using Equation (3) is called the raw transmission map, and is usually not good enough for highlight removal. This leads to an underestimation of the transmission of the highlight region and an overestimation of the transmission of the normal illuminance area. When an image contains low-light area and specular reflection area, if we directly use the raw transmission map to recover the specular reflection image, the removal of highlights is not obvious, and the low illumination area of the image will darken and lose details. Therefore, the highlighted area should have a low transmission value, and the normal and low illumination areas should have a high transmission value. Therefore, we designed a nonlinear function to refine the raw transmission map.
First, we define the adjustment factor wt as follows: where t is the raw transmission map, and the second adaptive constraint obtained as After constraining the transmission map, the transmission values of the highlight area decrease, signifying that the highlight component is better removed. The transmission values of normal and low illumination areas are increased to avoid pixel dimming.
Combining Equation (6) with Equation (8), the complete constraint on the transmission map t is:

C. REFINEMENT
Edge-preserving filters are classified into two types: local and global. Local filters such as bilateral filters and guided filters have the drawback of removing halos which are completely at the edges [34][35]. On the other hand, global filters such as weighted least squares (WLS) and L0 gradient minimization filters have the advantage of suppressing halos at the price of global intensity shifting [36].
For specular images with large highlighted pixels, guided filtering is only applicable to local computation and cannot eliminate the clustering phenomenon of pixels. The processed image still has a halo effect, so we use an L0 gradient minimization filter to refine the transmission map after adaptive constraints.
In the previous part, we obtained the transmission map t with two adaptive constraints, and then used the map as the input for the L0 gradient minimization filter. The gradient value of the output is considered according to the input image t. Initially, the input t is assigned as s, and then using Equation (10) where λ is the smoothing parameter and η initially equals η0. Then, the output s is calculated using Equation (11).
where F indicates the fast Fourier transform operator, F* indicates the complex conjugate operator, and F (1) indicates the Fourier transform of the delta function. This is repeated for a certain number of iterations until η equals ηmax. In this case, λ = 0.005, η = 2 and ηmax = 10 5 . VOLUME XX, 2017 9 After refinement, the output of the L0 gradient minimization filter is taken as the final transmission map t , and the scene radiance J is restored as follows: ( , ) (1 ( , )) ( , ) ( , )

D. NONUNIFORM ILLUMINATION COMPENSATION
We propose a nonuniform illumination compensation method to compensate for scene illumination and avoid scene darkening for the recovered image obtained by Equation (12). Gamma correction is widely used in low illumination enhancement and fog removal [37][38]. For some underexposed and overexposed images, gamma correction recovers the information well. For an overall dark or bright image, a relatively satisfactory effect can be obtained by adjusting the parameters.
General gamma correction refines the input image by a power index γ. For a grayscale image I, gamma correction transforms each pixel I(x,y) into the output O(x,y) according to the following rule: ( , ) ( , ) 255 255 Specular reflection mostly occurs under low illumination, and the saturation and brightness of the local area in the image are normally too high. Ordinary gamma correction will reduce or improve the brightness of the image as a whole, but will lose details in the process. Therefore, we consider local gamma correction [39], where the exponent of local gamma correction will not be a constant but instead will be chosen as a function that depends on the point (x,y) to be corrected and on its neighboring pixels N(x,y). Equation (13) thus becomes: First, we convert the image from the RGB space to YUV space to obtain the brightness component Y of the image. To preserve image details, the edge-preserving smoothening operation is applied on Y by using guided filter to obtain the final scene illumination Y0.
For the specular reflection image, to enhance the information of the highlight area without losing the information of the ordinary illuminance, γ should be a value of 1 or less than 1 for the normal and low illuminance areas, and a large value for the highlighted region. Therefore, we combine the inverse trigonometric function and gamma correction to construct a new illumination compensation function. For point (x,y), the gamma correction value obtained is: 0 ( , , ( , )) 2 arctan( (1 ( , ))) x y N x y b a y x y c where y0=Y0/255; γ usually taken as 0.5, since the value of b is usually between 0 and 0.5. The lower the ambient illumination is, the higher the value of b is set to enhance the ambient brightness of the image. c adjusts the gamma correction amplitude, and is usually set to 0.1. a is a stretching operator, and is used to adjust the stretching amplitude of the highlight image. The value of a is calculated as follows: where mean is the mean operation. The larger the mean value of the specular image illumination is, the smaller a value will be, and the greater the stretching degree of the highlight region will be, which can display more information in highlight areas. Applying our gamma correction function to Y, the enhanced illuminance component Y is obtained:  As shown in Fig. 2, when b=0, the value of γ is approximately 1 for areas with ordinary brightness, and the brightness component does not change. The highlighted areas have a large γ value, which can be well stretched to show the details.
When the image is converted from the YUV space to the RGB space, to compensate for the change in saturation, the following formulation is applied to the RGB channels. The transformed values R , G , and B are obtained as follows:

III. EXPERIMENTAL RESULTS AND DISCUSSIONS
In this section, we make a subjective and objective evaluation of the enhanced specular reflection image. To verify the performance of the proposed method, we compare the proposed method with existing methods, including Akashi et al. [40], Yamamoto [12], Fu et al. [17], Saha R et al. [26], Zhu et al. [29], and Xin [25]. The experimental results of this paper consist of four parts. In Part A, six comparison methods are introduced, and the enhanced specular reflection image is subjectively evaluated. In Part B, three evaluation indicators are used, and the processing effects of other algorithms are objectively evaluated. In Part C, we conduct an ablation experiment to prove the effectiveness of the proposed strategy, and discuss the effect of our adaptive constraint and nonuniform illumination compensation. In Part D, we discuss the time complexity of the algorithm. To prove the effectiveness of the proposed method, all compared approaches were implemented in MATLAB 2018a on an Intel Core i5 3.20-GHz processor with 8 GB of RAM, running a Windows 10 operating system.

A. SUBJECTIVE EVALUATION
To verify the authenticity of the method, Fig. 3 to Fig. 5 are all specular reflection images taken randomly in real-life scenes. Fig. 6 is an image contains both high illumination and low illumination areas, which collected from [4]. By observing the experimental image, it can be found that the presence of highlights in the original image makes some areas of the image very bright, which not only obstructs the information of the partial areas of the image but also changes the color of the image surface. Figs. 3(b)-(e)-6(b)-(e) show that Akashi et al. [40], Yamamoto [12], Fu et al. [17], and Saha R et al. [26] did not recover the information of the area blocked by the highlight. This is because the above algorithms have certain requirements for the image when processing specular reflection images, and most of the algorithms use the estimation of highlight components to achieve separation. Because the estimation of the specular component is not accurate, the image cannot obtain satisfactory results. Specifically, the dichromatic reflection model-based methods often wrongly detect white text texture as highlight, since they fail to semantically distinguish highlight regions from white material surfaces. Compared with the algorithms in [40], [12], [17] and [26], the method proposed in this paper can recover the information of the highlight area in the image very well. As shown in Figs. 3(f)-6(f), due to image fusion techniques, Zhu et al. [29] effectively solves the blur of highlight and improves contrast. By observing the window in Fig.6(f), we find the algorithm has defects in image restoration of the brightest area. As shown in Figs. 3(g)-6(g), Xin [25] uses an improved dark channel prior to accurately estimate the specular component to recover the specular reflection information well. However, the brightness of the image is low, resulting in information loss and poor image quality. As seen in Figs. 3(h)-6(h), our method can not only enhance the information of areas affected by highlights in the image but can also enhance the information of areas with low illumination to achieve the optimal visual effect. The image processed by the algorithm proposed in this paper retains more detailed features compared with other algorithms. After processing, the information in the highlighted area of the image is clearly discernible, and the contrast, sharpness and color features of the image are significantly enhanced.

B. OBJECTIVE EVALUATION
Objective evaluation indicators are usually divided into two categories: reference methods and nonreference methods. Because we did not have true and reliable specular-free reference images, to test the effect of each algorithm more comprehensively, we used three nonreference methods, the information entropy H, image edge intensity v and natural image quality evaluator (NIQE). To quantify the comparison of the algorithms, we performed a qualitative no-reference quality assessment with 200 specular reflection images. Note that higher evaluation values H and v represent better recovery effects, while lower evaluation values NIQE represent better recovery effects. We compare Akashi et al. [40], Yamamoto [12], Fu et al. [17], Saha R et al. [26], Zhu et al. [29], Xin [25] and our proposed method. The results are shown in Table 1-4. As shown in Table 1, H was taken as the evaluation metric, the results of the proposed method were better than those of the other methods, which showed that our method can effectively reveal the details in both low-light and highlight areas of an image.
As shown in Table 2, v was taken as the evaluation metric, the results of the Saha R et al. [26] and Zhu et al. [29] were better than the proposed method because the method of Saha R et al. [26] used low-light enhancement technique [27] and the method of Zhu et al. [29] used contrast limited adaptive histogram equalization algorithm (CLAHE) to adjust the brightness of the image, however two approaches led to increased time complexity and increased the noise. Table 3, NIQE was taken as the evaluation metric, the results of the proposed method were smaller than those of the other methods, which showed that the quality of image processed by our method were better than other methods. The processing data presented in Table 4 further demonstrates the superiority of the proposed algorithm.

C. ABLATION STUDY
In this part, we discuss the effect of adaptive regulation and nonuniform illumination compensation. In this study, three combinations are conducted for ablation study, namely S1,   S2, S3, where S1 represents only the color attenuation prior In this part, we discuss the effect of adaptive regulation and nonuniform illumination compensation. In this study, three combinations are conducted for ablation study, namely S1, S2, S3, where S1 represents only the color attenuation prior method, S2 represents our adaptive adjustment combined with the color attenuation prior method, and S3 represents our method. As shown in Fig. 7(b), the direct use of color attenuation prior technology causes the low illumination area to darken and causes black pixels to appear, while the highlight component in the highlight area cannot be completely removed. The license plate in the image is still blurry, black pixels appear on the ground. Fig. 7(c) shows that the application of our adaptive adjustment enhances the information in the areas affected by highlights and protects the information in the nonhighlighted areas. The license plate in the image was clear to identify. Fig. 7(d) displays that after our adaptive adjustment and nonuniform illumination compensation, the highlight area is further compressed, the information is more obvious and the low-illumination area is also enhanced. The image is enhanced in terms of contrast and color.
We evaluated the effect of the use of adaptive constraints and nonuniform illumination compensation on the results, the average values of H, v and NIQE for Fig.3 to Fig. 6 has been shown. Table 5 shows that the use of adaptive constraint technology causes a 1.2% rise in H, 13.2% rise in v and 5.2 decrease in NIQE. Adaptive constraint technology can better restore the edge, improve contrast, avoid the appearance of black pixels, and that several indicators are improved. On the other hand, in Table 5, there is 4.3% and 23.8% rise in H and v, meanwhile 7.8% decrease in NIQE. Our nonuniform illumination compensation strategy increases the brightness and saturation, resulting in an increase in all three indicators. Therefore, it can be concluded that the nonuniform illumination compensation strategy can significantly enhance the specular reflection image.

D. TIME COMPLEXITY
As shown in Fig.1, our method mainly includes four steps, calculating transmission map using color attenuation prior, two adaptive constraints for transmission map, artifact removal, and nonuniform illumination compensation strategy. Therefore, the time complexity of the proposed algorithm is the sum of these processes. Let n represents the total number of pixels of the input image, c represents the number of iterations of L0 gradient minimization filter, and rm represents the size of the block window in the guided filtering algorithm. The time complexity of each process in the proposed algorithm is shown in the following table. Because c and rm are much smaller than n, as shown in Table 6, the time complexity of the proposed algorithm is O(n 2 ). For an image of size 500 × 600, the running times of the methods of Akashi et al. [40], Yamamoto [12], Fu et al. [17], Saha R et al. [26], Zhu et al. [29], Xin [25] and our proposed method are shown in table 7. The computation time is obtained by averaging the testing results of ten times. By observing the data, we find that the time complexity of the proposed algorithm is lower than the time complexity of the methods of Akashi et al. [40], Yamamoto [12], Fu et al. [17], Saha R et al. [26], Xin [25] The proposed method requires more time than the methods of Zhu et al. [29], but our results are the best in terms of subjective and objective aspects.

IV. CONCLUSION
The main purpose of this study is to enhance the images affected by strong light in real-life scenes, especially information in images, and to improve the overall quality of the images to meet the needs of subsequent computer vision algorithms and practical applications. Therefore, an effective algorithm was proposed for specular reflection image enhancement. First, the specular reflection image was preliminarily processed by a color attenuation prior, and then two adaptive factors were set to adjust the raw transmission map to better remove the specular component. The L0 minimum gradient filter was used to refine the transmission map to eliminate some artifacts, such as noise and halo color clustering, and nonuniform illumination compensation was used to adjust brightness and enhance the saturation of the image. The experimental results show that this method effectively improve the clarity of the image and enhance the information of the specular reflection image compared with traditional methods. A single low dynamic range image can be restored to a high dynamic range image, and visually, the proposed method provides a more pleasant output than other methods.