Color Correction Based on CFA and Enhancement Based on Retinex With Dense Pixels for Underwater Images

Color correction and enhancement for underwater images is challenging due to attenuation and scattering. The underwater images often have low visibility and suffer from color bias. This paper presents a novel color correction method based on color filter array (CFA) and an enhancement method based on Retinex with dense pixels and adaptive linear histogram transformation for degraded color-biased underwater images. For any digital image in the RGB space, which is captured by digital camera with CFA, their RGB values are dependent and coupled because of the interpolation process. So we try to compensate red channel attenuation of underwater degraded images from the green channel and blue channel. Retinex model has been widely used to efficiently handle low brightness and blurred images. The McCann Retinex (MR) method selects a spiral path for pixel comparison to estimate illumination. However, the simple path selection doesn’t include global light dark relationship of the whole image. So we design a scheme to gain much well-distributed and denser pixels to obtain more precise intensity of illumination. Besides, we design a piecewise linear function for histogram transform, which is adaptive to the whole RGB value. Experiments on a large number of underwater degraded images show that, the processed images by our method have clearer details and uniform visual effect for all channels in RGB color space and our method can also obtain good performance metrics.


I. INTRODUCTION
Underwater vision plays an important role in ocean resources exploration and engineering [1], [2]. Due to wavelength-dependent and selective light absorption, underwater images always suffer from color castes and look bluish. When the distance from the imaging scene to the camera is being increased, the red channel will disappear first. The red channel map is darkened, and the value of the pixels in the red channel becomes small. In this regard, the color of such image should be corrected. Moreover, the scattering of light makes the contrast relatively low. Thus, the contrast of underwater images is often unsatisfactory [3], [4].
The associate editor coordinating the review of this manuscript and approving it for publication was Senthil Kumar.
The underwater imaging theory and underwater images enhancement or restoration methods have been widely studied these years [5]- [7]. Retinex-based methods and histogram transform based algorithms are commonly used for underwater image enhancement. Iqbal et al. [8] performed contrast stretching in RGB color space and saturation and intensity stretching in HSI color space to enhance underwater images. They also proposed an unsupervised color balance method to improve contrast in RGB color space and in HIS color space [9]. However, their method failed in turbid environment, for not considering the influence of scattering. Ancuti et al. [10] fused a color compensated image and white-balanced one from the original degraded underwater image for enhancement. However, this method is still unable to generate satisfactory results when the red channel is severely attenuated. Ancuti et al. [11] assumed that green channel is the counterpart of red channel, and they compensate the attenuation of red channel and blue channel from that of the green channel. Fu et al. [12] developed a three steps Retinex-based variational framework. Ghani et al. [13] presented an integrated color model by forcing the stretched images in RGB color model to follow the Rayleigh distribution. Besides, they combined global and local contrast stretching to increase underwater image quality [14]. Huang et al. [15] proposed a simple strategy for shallow-water image enhancement by adaptively obtaining the parameters. Li et al. [16] corrected color distortion by defining a color transfer function and using a generative adversarial network (GAN) to accomplish optimization. Li et al. [17] proposed a color correction GAN, which takes raw unlabeled underwater images as input, and outputs restored ones. Emberton et al. [18] detected and segmented regions without haze, and then estimate illumination by white balancing approach. Gao et al. [19] proposed an underwater image enhancement model inspired by the morphology and function of the teleost fish retina. Serikawa et al. [20] proposed a fast joint trigonometric filtering defogging algorithm. Galdran et al. [21] made improvements from the point of view of the dark channel prior [22], and proposed a suitable model to highlight the red channel, and some other improved methods based on the dark theory were also proposed in [5], [23]- [27]. Zhao et al. [28] combined underwater optical models and the specific properties of background light. Lu et al. [29] proposed an underwater imaging model to tackle the attenuation error. Lu et al. proposed two methods based on deep learning [30], [31]. These two methods both achieved good results but their application are restricted more or less by lack of training data. Li et al. [32] enhanced underwater images by removing color cast and restoring visibility. They also proposed an effective visibility recovery algorithm based on the principle of the minimum information loss of the three color channel and the inherent relationship of the transmission graph [33] and a hybrid method to correct underwater images [34]. Peng et al. [35] estimated the depth of the underwater scenes by utilizing image blurriness and light absorption. Fu et al. [36] presented two-step method: an effective color correction strategy by pixel-wise linear transformation and an optimal contrast improvement method. Wang et al. [37] utilized a non-locally adaptive attenuation-curve prior and imposed some constraints on saturation. Peng et al. [38] calculated the color change and the difference between the observed intensity and the ambient light to estimate the scene transmission. Halimi et al. [39] presented two methods to jointly restore the depth map and reflectivity image. Chang et al. [40] solved the consequences of scattering and absorption by five major steps. Li et al. [41] constructed an underwater image enhancement benchmark and evaluated and the performance and limitations of stateof-the-art algorithms.
Much progress has been made on the restoration and enhancement for underwater images. However, little work has focused on color-correction for underwater images. In this paper, we present a novel color-correction method and an enhancement method based on Retinex with dense pixels and adaptive linear histogram transformation for underwater images. This paper introduces the following main contributions: 1) We successfully make color-distortion correction for the red channel from the other two channels, inspired the fact that the pixels of RGB images captured by color filter array (CFA) based digital camera are dependent and coupled by the interpolation process. 2) We accurately estimate the illuminance component by designing clockwise and counterclockwise paths from four diagonal of a square for the McCann Retinex (MR) method. 3) We also present a linear piecewise adaptive histogram transform algorithm to improve the visual quality of underwater images.

II. RETINEX THEORY
Retinex theory [42]- [48] provides an enhancement method for low light images based on the theory of color constancy of human eyes. Retinex theory mainly includes two aspects: the color of the object is determined by the reflection ability of the object to the long wave, medium wave and short wave light, not by the absolute value of the reflected light intensity; the color of the object is not affected by the non-uniformity of light, and has consistency. The imaging model for Retinex is: where (x, y) denotes the specific location of the pixel; S represents the captured image by the camera; L and R represent the illumination component of the ambient light and the reflection component of the target object carrying the image detail information, respectively. L and R are considered as independent, Retinex tries to estimate L from S to obtain R which is of interest and desired. On both sides of Equation (1), the logarithmic operation is taken to form a sum: Finally, we have Typical Retinex method includes two types: center/ surround Retinex, for instance, Single-Scale Retinex (SSR) [42], Multi-Scale Retinex (MSR) [43], Multi-Scale Retinex with Color Restoration (MSRCR) [44] and path-selection-based method, for example, McCann Retinex (MR) [49].
The enter/surround Retinex estimates l by the convolution s and some surround function f as follows: The widespread used surround function is Gaussian kernel: SSR uses one Gaussian kernel, while MSR utilizes several Gaussian kernels with different scales and the component l is estimated by: The scale parameter of Gaussian kernel has a great effect on enhancement and not easy to determine. MSRCR is expressed as follows: where c (x, y) is a color restore function used to adjust the percentage of three RGB color channels. It is also difficult to design an appropriate color restore function. MR selects a certain path to calculate the light and shade change on the path shown in Fig. 1 [49]. The distance from the target pixel is reduced by half during iterations. After the corresponding point is selected, the values of the pixels on the path are compared and updated. When the path covers the whole image, the illuminance component is estimated when through iteration. MR uses global illumination estimation for local enhancement, so when there is uneven illumination or high background brightness, the results are poor. Hence, we will design a better path selection strategy to obtain more precise results. Because in the HSV color space the hue component and the saturation component remain unchanged independent of the illuminance information of the image, it is usual to enhance the value component by the MR method to improve the uneven illumination of underwater images.

III. PROPOSED METHOD
Considering the characteristics of underwater imaging and the limitations of directly processing underwater images, we propose an underwater image enhancement method based on Retinex with dense pixels and histogram transformation. Our method is focused on the following steps: (1) Firstly, we make color-distortion correction for red channel from green channel and blue channel; (2) Secondly, classical white balance algorithm is used to further solve color cast of underwater images; (3) Then the image is transformed from RGB space to HSV space, the V component is processed by McCann Retinex (MR) algorithm with dense pixels to make its illumination become more uniform; (4) Finally, after the image is transformed back to RGB color space, it is adjusted by a piecewise linear function. Fig. 2 shows the whole flowchart of our method.

A. RED CHANNEL COMPENSATION FROM GREEN CHANNLE AND BLUE CHANNEL
Most RGB images are captured by digital camera with color filter array (CFA) [50]. Fig. 3 shows Bayer CFA, from which it can be seen that, for any specific pixel the image sensor obtains the intensity information of only one RGB color channel. The other two missing colors are calculated by demosaicing algorithm. Therefore, their RGB values are dependent and coupled because of the interpolation process. The pixel value of the red channel is closely related to the pixel value of the blue channel and the green channel in its neighborhood. Inspired by this fact, we try to compensate the worst red channel of underwater images from their relative better channels, i.e., green channel and blue channel. The compensation function for red channel is designed as follows: whereĨ R is the pixel value of the red channel after compensation; I R , I G and I B are the pixel values of red channel, green channel and blue channel of the original degraded underwater image, respectively; α ∈ (0, 1) is a constant;Ī G andĪ B are the average pixel value of green channel and blue channel within the local window of the original image with (x, y) as its center. The size of the window can be set 3 × 3 or 5 × 5. Fig. 4 gives the color corrected underwater images and their histograms by our method and that in Ref. [11]. From Fig. 4, it can be seen that our compensated images have better visual effects and more uniform histograms.

B. WHITE BALANCE
The process of removing color cast so that ''white remains white'' under the capture and viewing illuminants is termed white balancing. One means of performing white balance is to assume that a white patch induces the maximal response in one or more of the camera sensors in RGB channels. Then, the RGB values of white-balanced image are given by R/R max , G/G max , B/B max , respectively, where the subscript ''max'' means the maximal within the original whole image.   Here we utilize this simple white balance method to further improve color visual effect. Fig. 5 gives three raw underwater images and the compensated images and the white-balanced images. As shown   in Fig. 5(c), after the first two steps of our method, the restored images are satisfactory.

C. ILLUMINAACE IMPROVEMENT BY RETINEX WITH DENSE PIXELS
After the above two steps, the color deviation of underwater images have been greatly improved. However, when underwater images are obtained especially in deep water areas, artificial light sources are often needed as auxiliary light sources for imaging, which usually results in uneven illumination and blurred details of the images. However, they cannot be solved by correcting color deviation. So we further utilize MR method to enhance them.
It is obvious that the path of MR method mainly covers the upper right area of the image and does not completely cover the whole image. If the target object is in the lower left of the image, the accuracy of the illuminance component estimated by this path is poor. In order to overcome the shortcoming of the above path selection strategy, we design a much better path mode with dense pixels shown in Fig. 6. It can be seen that our designed path covers most areas of the image and the distribution of involving pixels is relatively uniform.
Although the above path has covered the whole image, it does not take into account the influence of the illuminance information of the pixels far from the starting point, so the enhancement effect is still poor for those images with large local pixel deviation. In order to overcome this deficiency, a modified MR method with dense pixels is further proposed, which is shown in Fig. 7. Our selected paths include both clockwise direction and counterclockwise direction, and the path can start at any diagonal of a square. In total, we have eight different paths. Finally the average value of the illuminance information obtained by these eight paths is calculated. Through this path selection method, the distribution of the pixels on the paths becomes uniform and dense, and in result the estimated illuminance component is more accurate. Fig. 8 shows restored underwater images by different Retinex methods: SSR, MSR, MSRCR and our MR method with dense pixels. In order to make comparison fair, all images are obtained directly from these methods, and neither of pre-processing or post-processing are included here. From  Fig. 8, it can be seen that our MR method outperforms. Fig. 9 shows restored underwater images after the first three steps of our method from the same degraded images shown in Fig. (5)(a). It can be seen that the restored images by our MR method look much clearer.

D. ADAPTIVE HISTOGRAM TRANSFORM
According to the Gray-World theory, the average value of the normalized RGB three-channel pixel value of the perfect color image is 0.5, so the average value of any channel in RGB color space is about 128. Furthermore, based on the statistics of 500 natural images, we have found that the average pixel value of any RGB channel of natural images is mainly distributed in the interval of [100, 140]. However, it is not the case for underwater degraded images. Hence we design a piecewise linear function for adaptive histogram transform to adjust the pixel values in order to upgrade underwater images' visual effect. According to the main distribution interval of the mean pixel value of RGB channels for natural images, we design a piecewise linear function for adaptive histogram transform as follows: where I c andĨ c are the pixel value of any channel c ∈ {R, G, B} before and after histogram transformation, respectively;Ī c is the average pixel value of the original image; and ε is positive integer. Gamma-correction [51] is a simple and effective enhancement method which outputs image byĨ c = (I c ) γ . Fig. 10 shows the final restored and enhanced underwater images by histogram stretching [15], gamma-correction with different values for γ and our adaptive histogram transform (c) UBL [35]; (d) ULA [56]; (e) GHS [15]; (f) GLN [57] (g) our method. method. It is easily seen that our method has satisfactory visual effect. Fig. 11 shows restored underwater images after all four steps of our method from the same degraded images shown in Fig. (5)(a).

IV. EXPERIMENTAL RESULTS
To verify the effectiveness of our method, a large number of raw underwater images mainly from Ref. [41] are used to test. We compared the proposed method with several novel underwater image enhancement and restoration methods proposed in recent years, such as, global histogram stretching method (GHS) [15], underwater image enhancement by dehazing (UIED) [33], image blurriness and light absorption (UBL) [35], underwater light attenuation prior for underwater image (ULA) [56] global-local networks and compressedhistogram equalization (GLN) [57]. The visual quality of the results of those different methods is evaluated subjectively and objectively. Besides, we use the swatch image to verify the color correction effect.
A. SUBJECTIVE ASSESSMENT Fig. 12 shows eleven raw underwater images from [41] which were captured under different underwater scenes and the VOLUME 8, 2020  [33]; (c) UBL [35]; (d) ULA [56]; (e) GHS [15]; (f) GLN [57] (g) our method. processed counterparts by different methods. From Fig. 12(a), it can be seen that these raw images have different background colors and some are turbid and the surface of objects are obviously foggy. From Fig. 12(b), it can be seen that the method in Ref. [33] can effectively improve the contrast and improve the brightness, but the enhancement effect isn't effective and some processed images are still severely weak in the red channel. From Fig. 12(c) and Fig. 12(d), it can be seen that the method in Ref. [35] and Ref. [56] have good performance, but they may cause color distortion. From Fig. 12(e), it can be seen that the method in Ref. [15] can make underwater images very clear, however, it cannot effectively correct the color deviation, and the contrast is still poor. From Fig. 12(f), it can be seen that the method in Ref. [57] has great performance when dealing with various types of underwater distorted images, and it can effectively correct the color deviation, but the processed images looks a little dark. On the contrary, our method can not only improve the contrast and correct the color deviation, but also can make the brightness distribution uniform so it can finally lead to a better visual effect.
To further compare the results for color recovery, we use three color-map board images which are captured underwater shown in Fig. 13(a) and seven color map boards taken by different cameras shown in Fig 14(a) from Ref. [58] for experiments. The corresponding ground truth for these color-map board images are shown in Fig. 15. Experimental results for different methods are shown in Fig. 14(b)-(g) and Fig. 15(b) shows restored images from seven original color map boards taken by different cameras by our method. We can easily find that our method can successfully and effectively recover underwater images' color and the restored images are clear enough.

B. OBJECTIVE ASSESSMENT
To further assess the proposed method, we select Entropy, NIQE [52] and IL-NIQE [53], UIQM [54], UCIQE [55] to objectively evaluate the results of our method. In the following, we will briefly explain these evaluation indicators one by one. Entropy represents the richness of the information contained in the measured image. It can be obtained by characterizing the aggregation characteristics of the gray distribution of the image. Natural Image Quality Evaluator (NIQE) defines a natural scene statistic model in space domain and then collects all statistical features from it [52]. Integrated Local NIQE (IL-NIQE) can completely gain local distortion artifacts by integrating multiple features derived from a local multivariate Gaussian model [53]. Underwater Image Quality Measure (UIQM) is also a comprehensive underwater image quality evaluation method, which combines chroma, sharpness, and contrast to quantify degradation of underwater images [54]. Underwater Color Image Quality Evaluation (UCIQE) combines chroma, saturation, and contrast to quantify degradation of underwater images [55].
To make the evaluation results more objective, we randomly selected 100 underwater images from [41]. Table 1 shows five average performance indices obtained by the methods in [15], [33], [35], [56], [57] and ours. Our method has the best results measured by Entropy, NIQE, IL-NIQE and UIQM and a good UCIQE result.
The results show that our enhancement method can greatly improve underwater images' visual effect and has excellent performance indices. Our method can improve the clarity, contrast and brightness of underwater images and it can also reduce color cast.

V. CONCLUSION
In this paper, we propose an underwater image color correction method and an underwater image enhancement method. For any RGB image captured by digital camera with color filter array (CFA), its RGB values are dependent and coupled because of the interpolation process. Inspired this fact, we make color-distortion correction for the red channel from the other two channels. We also design a scheme to gain welldistributed and dense pixels to reform the McCann Retinex (MR) method. Hence, we can obtain more precise illumination intensity. To further improving the visual quality of the whole image, we also design a piecewise linear function for adaptive histogram. We conduct sufficient experiments on a large number of degraded underwater images. Our method outperforms state-of-the-art methods in objective metrics. We also estimate our method by subjective assessment. The images processed by our method have clearer details, uniform visual effect, and better color-correction results comparing with state-of-the-art methods.

ACKNOWLEDGMENT
Many thanks are given to Dr. C. Li [41] for sharing data with their team and to the anonymous reviewers for their valuable suggestions.