Underwater Image Enhancement Based on Color Balance and Multi-Scale Fusion

Underwater light absorption and scattering lead to color deviation, low brightness, fuzzy details and low contrast of underwater images. In this contribution, an underwater image enhancement algorithm based on color balance and multi-scale fusion is proposed. Firstly, a color balance method is used to correct the image color. Then, based on an improved dark channel prior algorithm, the local contrast information of the color balanced image is used to derive two atmospheric lights and transmission maps, which are transformed into the images with enhanced contrast and brightness, respectively. Finally, the multi-scale fusion method is adopted to fuse the contrast enhanced image and the brightness improved image according to the weights. The proposed algorithm is compared with other underwater image enhancement algorithms in qualitative and quantitative evaluation. Experimental results show that the proposed algorithm can effectively eliminate color deviation, remove dark areas, and improve brightness and contrast of underwater images. The enhanced images generated by the proposed algorithm are superior to those generated by other algorithms in indicators of PSNR, SSIM, UIQM and UCIQE. Thus, the proposed algorithm is suitable for underwater image enhancement.


I. INTRODUCTION
I N RECENT years, with the increasing development of human resources, land resources no longer meet the needs of human beings. Many countries gradually focus on exploring ocean resources. Acquisition of clear underwater images is of great significant for underwater exploration. Underwater image enhancement technology plays an increasingly important role in exploration of marine resource, detection of underwater facilities, observation of ocean scenery and research of seabed organisms. However, light can be absorbed and scattered by water and underwater particles [1]. The degree of light attenuation varies with the colors of light, which makes color deviation and low brightness appear in underwater images. Moreover, light scattering leads to detail loss and contrast degradation of underwater images. Therefore, the underwater image enhancement technology is challenging due to serious degradation and low brightness of underwater images. Due to the complexity of underwater environment, traditional image enhancement algorithms are not suitable for underwater images. In order to effectively improve the clarity of underwater images, a variety of underwater image enhancement methods based on nonphysical model and physical model are developed [2]. He et al. [3] proposed a dark channel prior algorithm (DCP) to dislodge fog in images, which is based on the principle that light propagation in water is similar to that in fog. However, the degree of light attenuation varies with the colors of the light, resulting in serious color deviation. To improve the DCP algorithm, Drews et al. [4] proposed a new underwater dark channel prior algorithm (UDCP) based on the principle that red light has greater attenuation than blue and green light in water. However, this algorithm only calculates the values of dark channels in the blue and green channel. Iqbal et al. [5] proposed a method combining histograms with unsupervised color correction, but the color distortion problem is difficult to be solved. Zhang et al. [6] proposed an underwater image enhancement algorithm of extended multi-scale Retinex (MSRCR), which extends the retinal enhancement algorithm with color recovery to the Lab space to overcome the halo phenomenon. However, there are too many parameters to be adjusted in the algorithm, making it difficult to implement. Garg et al. [7] proposed an underwater image enhancement algorithm of adaptive histogram equalization with limited contrast, which can effectively enhance contrast. Nevertheless, it fails to remove the color deviation of images. Ma et al. [8] proposed a combination of the dark channel prior method and the gray world method to defog underwater image, but the brightness of obtained images is still low.
With the development of deep learning technology, more and more algorithms based on deep learning networks are proposed to enhance underwater images [9]. For example, Fabbri et al. [10] proposed an underwater image enhancement method based on the cycle generative adversarial network (Cycle GAN), but the training time is too long and the obtained images are of low clarity. Islam et al. [11] built a model based generative adversarial networks for real-time underwater image enhancement. This method quickly enhances underwater images by improving visual perception, but the color of enhanced images is degraded.
Recently, the multi-scale fusion has become a research hotspot in the field of underwater image enhancement. Ancuit et al. [12] proposed a white balancing based image enhancement method that generates a sharpened image and contrast corrected image, which are fused to obtain the enhanced image. Gao et al. [13] proposed a new image enhancement method to restore image color by fusing the contrast corrected image and sharpened image. Wu et al. [14] proposed an underwater image enhancement method based on generative adversarial network with multi-scale fusion. However, due to the dark environment in deep water, underwater images commonly suffer from low brightness. These multi-scale fusion methods fail to improve image brightness, leading to limitations in image enhancement. Therefore, in order to solve the degradation and low brightness problem of underwater images, an underwater image enhancement algorithm based on color balance and multi-scale fusion is proposed in this paper. Firstly, a color balance method is used to restore the image color. Then, an improved dark channel prior algorithm is designed to derive a contrast enhanced image and a brightness improved image, which are combined by the multi-scale fusion method. Finally, the qualitative and quantitative assessments are analyzed on the proposed algorithm in comparison with several state-of-the-art underwater image enhancement algorithms. The results show that the images processed by the proposed algorithm are of corrected color, clear details, and high contrast and brightness.

II. PROPOSED METHOD
The underwater image degradation is very complex due to light absorption and scattering. The flow chart of the proposed algorithm is shown in Fig. 1. Aiming at the color distortion problem of the underwater image, a color balance method is used to restore the image color. Then, two atmospheric lights and their transmission maps are derived by using the improved dark channel prior method with different patch sizes that are calculated in the inversed version of the color restored image. The atmospheric light with high contrast and clear details is generated by the smaller patch size, while the other with high brightness is generated by the larger patch size. Two restored versions of the image, the contrast enhanced image and the brightness improved image are generated by different atmospheric lights and transmission maps. Next, the Laplacian, saturation and saliency weights of the restored images are calculated and normalized. Finally, the Laplacian of the restored images and Gaussian of the normalized weights are blended by the multi-scale fusion to generate the enhanced image.

A. Color Balance
Water absorbs different colors of light in different degrees. Red light that possesses the longest wavelength suffers from more serious attenuation than blue and green light, making the color of underwater images tend to be blue and green [15].
In order to solve the problem of color distortion, affine transformation based on accumulated histogram distribution is applied to color channels R, G and B. It is assumed that the highest values of channels R, G and B in the image correspond to white, and the lowest values correspond to black. The maximum range of values of each channel can be stretched to [0, 255] by affine transformation. When the color balance method is performed on the pixels in each color channel, a percentage e% of the pixels with values lower than or equal to bottom are saturated to bottom, and a percentage e% of the pixels with values higher than or equal to top are saturated to top. There are N pixels clipped on the histogram of each color channel. Therefore, both bottom and top can be calculated by the cumulative histogram of pixel values at positions of N × e% and N × (1 − e%), respectively. Then, affine transformation is used to map the pixel interval [bottom, top] to [0, 255] as follows: where x is the input pixel value and f(x) is the output pixel value. Parameter e is derived as follows: where p λ (λϵ {R, G, B}) represents pixel values in each color channel of the image I. Fig. 2 shows that the gray distribution in red, green and blue channels of the color balanced image is more uniform than that of the original image. For underwater images, the proposed color balance method is an effective tool to recover colors.

B. Improved Dark Channel Prior
The proposed color balance method can solve the problem of color distortion. However, the problem of details loss and low contrast and brightness still exist due to underwater light scattering and absorption. Considering the light reflected by the underwater scene, the direct component C d represents the light that reaches the camera without being scattered, and the forward scattering component C fs represents the light that has been scattered at a small angle by suspended particles before entering the camera. The back scattering component C bs refers to the scattered light that reaches the camera without reflection from the underwater scene.
The underwater imaging model is described as follows: The forward scattering component C fs can be ignored because the camera is not far from the scene. C d and C bs are defined in (4) and (5), respectively.
where x is a spatial position of the scene, J(x) is the scene light, A Ý is the atmospheric light, t(x) is the transmission map that indicates the light entering the camera directly, d(x) represents the scene depth, and μ represents the attenuation coefficient.
In underwater environment, μ contains both the scattering coefficient β and the absorption coefficient α [16]. Therefore, underwater images suffer from both low contrast and brightness. Based on the Robby's [17] law, a low-contrast and low-light underwater image I can be described by the following equation: where x is a spatial position of the image. To restored underwater images, it is essential to calculated t(x) and A Ý . The DCP method proposed by He et al. [3] is widely applied to calculate A Ý and t(x), where the dark channel is defined as follows: where x and y are the spatial positions of the dark channel and the local image, respectively. J c is the scene radiance for channels R, G, B, and Ω(x) is a local patch centered at x. Each pixel in J dark is of lower intensity than other pixels inside the patch Ω in J c . The atmospheric light A Ý can be obtained from the brightest pixels in the dark channel. Considering the influence of lighting sources outside patch Ω, the patch Ψ with twice the size of Ω is used by Ancuti, et al. [18]. Therefore, the local atmospheric light A c L∞ (x) inside the patch Ψ is defined as follows: where x, y and z are spatial positions of the local image, min z Ω(y) I c (z) and inside patch Ω, respectively. min z Ω(y) I c (z) denotes the lowest intensities of each channel inside the patch Ω and A Ý can be derived by calculating the highest intensities inside the patch Ψ. Once A Ý is obtained, dividing both sides of (6) by A Ý and using the dark channel in (7), the transmission map t(x) is calculated as follows: The DCP method is an easy and effective way to restore degraded images. However, owing to light scattering and absorption in water, the degradation of underwater images is complex, making properties differ in image areas. Therefore, the restored images formed by DCP may change a lot when selecting different sizes of patches Ω based on properties. Small size patches Ω are suitable for areas with steady intensities, but make the image color excessively saturated and fail to remove dark image areas. Large size patches Ω are suitable for areas with variable intensities, but cause halos that make details of the image fuzzy [19]. The DCP method adopts a given patch Ω, which derives only one atmospheric light and transmission map, resulting in overly-bright or washed-out restoration when using this method in deep underwater scenes.
To solve the limitation of the DCP in underwater scenes, an improved DCP is proposed. Considering the attenuation coefficient in (5), which consists of coefficient β and coefficient α that cause low-contrast and low-bright effect in underwater images, respectively. This paper uses a small size patch and a large size patch to derive two different atmospheric lights and transmission maps. Then the two restored images, a high-contrast one and a high-bright one, are obtained and fused to form the final enhanced image.
To obtain the ideal patch sizes for calculating the atmospheric light, this paper considers the intensity change due to its value reflects the contrast information inside an area. The size of patch Ω is S Ω ×S Ω , and S Ω is defined as follows: where σ represents the standard deviation of the intensities in each channel of the image, and m is a multiplication factor that determines the radius of influence of each light source, which needs to be gradually restricted according to the candidate size of the patch. The principle is that larger patches are used to derive brighter atmosphere lights, while the smaller ones have greater impact on contrast enhancement. Coefficient β and coefficient α decrease the contrast and brightness of images, respectively. The DCP method derives only one atmosphere light and transmission map, which is not sufficient to restore images. This paper uses two different m values to derive two atmosphere lights, which can be defined as follows: The size of patch Ψ is twice as the size of patch Ω, which is also determined by m. A high m value generates the A c LCG∞ with high brightness, while a low m value generates the A c LCG∞ with clear details and high contrast. This paper derives two A c LCG∞ : one is derived by using m = 5 and the other is derived by using m = 20. Using m = 5 is to enhance the details and contrast of the image and using m = 20 is to improve the brightness of the image.
Once the two A c LCG∞ with different m values are obtained, their corresponding transmission maps can be calculated by (9). However, noise that appears in the process of transmission maps forming causes the problem of loss of image edges. To recover the image edges, the two transmission maps are filtered by the Fast Guide filter [20]. The Fast Guide filter is an edge preserving filter defined in (12).
where k represents the index of a local square window w with the radius of 16, i represents the index of a pixel, and q i is output image pixels generated by the filter based on input image pixels p i . The input image is sub-sampled by the ratio of 4. Coefficients a k and b k are calculated in (13) and (14), respectively.
where μ k and σ 2 k represent the mean and variance of pixel intensities inside w k of the input image p, respectively, and |w| is the number of pixels inside w k . The input image p owns an average pixel intensityp inside w k and eps determines the smoothness of the filtered image. Here, eps is 0.45.
The transmission maps filtered by the Fast Guide filter are used to restore images by (15).
where t 0 = 0.02 is the lower bound to t(x, m). The color balanced image and its color channel histograms are shown in Fig. 3(a)-(d). The restored image generated with m = 5 and its color channel histograms are shown in Fig. 3(e)-(h), and the restored image generated with m = 20 and its color channel histograms are shown in Fig. 3(i)-(l). It is shown that the image generated from the atmospheric light with m = 5 has higher contrast and more uniform histograms of channels R, G and B, while the brightness of the image generated from the atmospheric light with m = 20 is significantly enhanced and the gray levels of R, G and B channels of the image are also significantly improved.

C. Multi-Scale Fusion Technology
The two obtained images need to be fused to produce a better enhanced one by preserving most important features of them. The weights contain the information and features of the image, which are usually adopted in fusion process. In practice, no weight map can represent all the significant features of the image. In this paper, the Laplacian weight, saturation weight and saliency weight are selected, which can guarantee that areas of the input images with high contrast or brightness receive higher values. To better describe the spatial relationship between degraded areas, the weights are designed in the pixel form.
Laplacian Weight: The Laplacian weight measures the global contrast information of the image and assigns high values to the areas with edges and texture. This weight is calculated by the absolute value of the Laplacian filter applied on the luminance channel as follows: where x is the pixel in the image and L lap (x) represents the value of the luminance channel filtered by the Laplacian filter. The Laplacian weight is responsible for highlighting areas with high intensity variation. However, this weight is not sufficient to recover the local contrast of the flat regions [21]. Therefore, two additional weights: the saturation weight and saliency weight, are also adopted in the fusion process. Saturation Weight: Since more saturated colors have higher values in the color channels, the saturation weight processes the color information by emphasizing areas with high degrees of saturation. To obtain this weight, the standard deviation between the input luminance and each color channel is calculated at each pixel position. The saturation weight can be calculated as: where R(x), G(x) and B(x) are color channel values, and L(x) is the luminance channel value. In this way, high values are assigned to areas with high degrees of saturation and small values are assigned to other areas. Saliency Weight: The saliency weight can distinguish important areas from their neighboring areas and highlight underwater objects that are unnoticeable. The main data information of the image concentrates in important areas. The weight is calculated as: where I a represents the average value of the input image, and I gau (x) represents the value of pixels of the input image filtered by the Gaussian filter. Normalization of the Weights: To generate consistent results, the three weights W L (x), W S (x) and W Sa (x) are combined into a normalized weight for each input image. The normalized weightW j (x) is calculated for each input as follows: where i represents the index of the three weights, and j represents the index of the input images. The normalized weights of the two input images are shown in Fig. 4. A single image fusion method used for underwater images can cause serious halos. In this paper, Gaussian pyramid and Laplacian pyramid algorithm are used to decompose the images, and then the decomposed images are fused by the multi-scale fusion method.
Gaussian Pyramid: The Gaussian pyramid consists of a set of images obtained by down-sampling the image. Before sampling, Gaussian filtering is required for the image. The equation G L for each layer of the Gaussian pyramid is defined as: where L is the index of the pyramid layers, N = 5 represents the total number of pyramid layers, w(m, n) represents a Gaussian kernel convolution with a size of 5×5, and Down() means that the filtered image is sampled by the factor of 1/2 in two directions. Laplacian Pyramid: The equation of P L for each layer of the Laplacian pyramid is as follows:  where Up() means that the filtered image is sampled by the factor of 2 in two directions. Fusion Process: The image fusion process is shown in Fig. 5. Firstly, the normalized weightsW j (x) and input images I j (x) are decomposed into Gaussian pyramid and Laplacian pyramid, respectively. Then the Laplacian input images and the Gaussian normalized weights are mixed at each level to produce the fused pyramid. Finally, the final enhanced image F(x) is obtained by summing up all levels of the fused pyramid: where l represents the index of the pyramid levels, and j represents the index of input images.

III. EXPERIMENTAL RESULTS AND ANALYSIS
To evaluate the proposed algorithm, the test underwater images are selected from multiple authoritative image datasets [22]. Qualitative and quantitative comparisons are conducted against five advanced underwater image enhancement algorithms, which includes the classical underwater dark channel prior algorithm (UDCP) [4], Zhang's multi-scale Retinex extension algorithm [6], Garg's contrast histogram equalization algorithm [7], Ma's combination of dark channel prior and gray world algorithm [8], and Islam's fast underwater image enhancement for improved visual perception algorithm (FUnIE-GAN) [11].

A. Qualitative Evaluation
In order to verify the effectiveness of different algorithms on color restoration, the underwater color card image is selected for experiment. The results are shown in Fig. 6.
As shown in Fig. 6, the image processed by UDCP algorithm has low brightness and serious color deviation. Zhang's algorithm significantly improves the brightness of the image, but the red channel is over enhancement and the contrast between adjacent blocks such as the yellow block and light green block is reduced. Garg's algorithm increases the contrast and distinction between adjacent color blocks, but it causes distortion in the purple and light coffee block. Ma's algorithm can accurately correct the color blocks, but fails to improve the brightness of the image. FUnIE-GAN algorithm is unable to improve the distinction and contrast between adjacent color blocks such as the light green and yellow block. Besides, this algorithm causes color distortion in the light blue block, orange block and brown block. It can be seen that the proposed algorithm improves the clarity and brightness of the image. Furthermore, the distinction between different color blocks and the contrast between adjacent color blocks are both improved. Thus, the color of the generated image is close to that of the standard color card. It shows that the proposed algorithm has a good performance in color recovery.
Next, images with different degrees of degradation are processed by each algorithm, as shown in Fig. 7.    Fig. 7, UDCP algorithm introduces color degradation and deteriorates the clarity of the images. Zhang's algorithm can effectively restore the image color, but the processed images are of low contrast and unclear details. Garg's algorithm can improve the contrast of the images. However, it brings about color deviation. The images produced by Ma's algorithm are of low brightness, resulting in poor visibility of the whole images. FUnIE-GAN algorithm brings pseudo shadows into the processed images, resulting in low clarity of the images. The proposed algorithm shows high robustness in enhancing the underwater images with different degrees of degradation. This algorithm can effectively remove color deviation and improve the contrast of the images, making the details of the images clearer than those of other algorithms. Therefore, the processed images can yield a better visual effect that is consistent with the human eyes.

B. Quantitative Evaluation
To objectively evaluate the performance of different algorithms, this paper selects four image quality metrics: the peak signal to noise ratio (PSNR), structure similarity index measure (SSIM), underwater image quality measure (UIQM) and underwater colorful image quality evaluation (UCIQE).
As shown in Table I, except Image1 processed by Ma's, the proposed algorithm obtains the highest PSNR values, which means less image noise and more image information. As shown in Table II, the proposed algorithm obtains the highest SSIM values in all images, which can generate the enhanced image of a more complete structure and more valuable information. As shown in Table III, except Image1 processed by FUnIE-GAN and Image6 processed by UDCP, the proposed algorithm obtains the highest UIQM values, which proves that the proposed algorithm can produce a better enhancement in terms of colorfulness, sharpness and contrast. As shown in Table IV, the proposed algorithm obtains the highest UCIQE values in all images, which achieves a better effect in balancing tone, saturation and contrast. The results show that the proposed algorithm has a better enhancement effect than other algorithms in the indicators of PSNR, SSIM, UIQM and UCIQE.

IV. CONCLUSION
The underwater environment is too complex, which causes underwater images suffering from color distortion, low contrast, and detail loss. This paper proposes an underwater image enhancement algorithm based on color balance and multi-scale fusion. Firstly, a color balance method is used to restore the image color. Then, an improved dark channel prior method is used to obtain a contrast enhanced image and brightness improved image. Finally, the two images are combined by the multi-scale fusion method according to the weights. Both qualitative and quantitative evaluation demonstrate that the proposed algorithm can better correct color, enhance details, remove dark areas of the image, and improve contrast and brightness of the underwater images compared to other underwater image enhancement algorithms.