Underwater Image Restoration Using Geodesic Color Distance and Complete Image Formation Model

Underwater images suffer from different types of quality degradation, including haze, blur, low contrast, and color distortion, owing to light scattering and absorption. This article proposes a novel underwater image restoration algorithm based on the complete underwater image formation model (UIFM). Although the majority of the existing methods consider the direct transmission and backward scattering components only, this study, in addition, includes forward scattering in the UIFM. We estimate the transmission map based on the observation that the scene distance is inversely proportional to the geodesic color distance from the background light. We also approximate the point spread function in the forward scattering term to estimate the scene radiance more faithfully. Moreover, we obtain the optimal parameters of the UIFM required for transmission estimation and scene radiance restoration by minimizing a cost function composed of the sharpness, information loss, and dark background prior. The experimental results confirm that the proposed algorithm considerably improves the quality of the estimated transmission maps and restores scene radiance compared with the existing state-of-the-art methods.


I. INTRODUCTION
Underwater imaging systems are used in numerous applications, e.g., monitoring underwater environments, construction of underwater artificial facilities, and searching for and rescue of sunken ships. In addition, people capture underwater images with waterproof mobile cameras for entertainment purposes while scuba diving or snorkeling. However, extreme underwater environments often degrade the quality of the underwater images compared to that of ordinary images captured in the air. Accordingly, image processing and computer vision methods yield poor performance when applied to such quality-degraded underwater images.
As shown in Fig. 1, light absorption and scattering are two main causes of quality degradation in underwater images. The ambient light illuminating the underwater scene is typically regarded as the sunlight attenuated by the water. The scene radiance, the reflected light from the surface of underwater scene, is also attenuated by water while traveling toward the The associate editor coordinating the review of this manuscript and approving it for publication was Wei Zhang. camera. The attenuated scene radiance is called the direct transmission component (DTC), where the ratio of the DTC to scene radiance is the transmission. Moreover, the amount of light attenuation varies according to the wavelength of the light such that red is more rapidly attenuated than blue or green. Both the ambient light and attenuated scene radiance cause color distortion such that that we typically see bluish or greenish underwater images. The direction of the DTC is perturbed in small angles generating the forward scattering component (FSC), which is a blurred version of the DTC. Moreover, the scattered ambient light is observed as the backward scattering component (BSC), which covers target underwater scenes and generates a haze effect. Owing to the blur and haze, underwater images, in general, yield low contrast. Figs. 1(c-f) display typical quality-degraded underwater images where we can observe blur, haze, low image contrast, and color distortion.
From the literature, there has been other research for underwater image restoration or enhancement. Underwater image enhancement methods basically attempt to provide visually pleasing images [1]- [4], and thus cannot recover the original scene radiance correctly. McGlamery [5] proposed an underwater image formation model (UIFM) composed of the DTC, FSC, and BSC. Jaffe [6] also revised the FSC in the UIFM. Underwater image restoration methods estimate the original scene radiance from an input image based on UIFM. However, it is difficult to determine all the related parameters in the UIFM completely, and hence the existing methods have typically used simplified UIFMs by removing either BSC or FSC from UIFM. For example, Trucco and Olmos-Antillon [7] ignored the BSC assuming underwater images captured in shallow water, and Schechner and Karpel [8] omitted the FSC by investigating the amount of degradation associated with the BSC and FSC. In practice, the majority of the existing underwater image restoration methods [8]- [14] adopt a simplified underwater image model (sUIFM) that considers the DTC and BSC only.
Note that sUIFM is equivalent to the image formation model used for dehazing [15], [16]. Therefore numerous underwater image restoration methods have been inspired by ordinary image dehazing methods where the unknown parameters of the background light and transmission map are mainly estimated. For example, the background light ahs been estimated as the brightest pixel color searched using quad-tree subdivision schemes [17], [18]. Moreover, the background light of underwater images can also be estimated more reliably by considering the characteristics of the underwater images such as color difference [19] or blurriness [20]. Regarding transmission estimation, Chao and Wang [10] applied the dark channel prior (DCP) [21] such that the intensity in the dark channel of a haze free image is almost zero. Drews et al. [12] modified the DCP by considering only the green and blue channels as candidates of the dark channel. A non-local prior [22] has also been applied to underwater image restoration [13], [14] assuming that the pixels in a haze free image are composed of several color clusters; however, a hazy image exhibits lines in color spaces that connect the color clusters to the background light color. A physical light degradation model based on the relative distance between the target scene and camera has been used to estimate the transmission map of an underwater image [9], [23]. Carlevaris-Bianco et al. [9] estimated the transmission based on the maximum intensity of the red channel prior such that the absorption rate of the red color was greater than that of the green and blue colors. Peng et al. [23] proposed the blurriness prior for transmission estimation such that the captured scene becomes more blurred at farther distances.
Conversely, attempts have been made to reliably estimate the attenuation coefficients of the UIFM for underwater image restoration. Although the attenuation coefficients in the image formation model for ordinary hazy images captured in the air are the same across the color channels, those for underwater images are different based on the color channels [24]. Chiang and Chen [11] estimated the coefficients for the three color channels using the statistical ocean characteristics categorized in [25]. Zhao et al. [26] derived the ratios of the attenuation coefficients by assuming that the ambient light is constant across the color channels. Berman et al. [13] obtained the sets of the coefficient ratios measured for 10 water types, and selected the best set of the ratios satisfying the Gray World Assumption [27]. Akkaynak and Treibitz [28] estimated different attenuation coefficients associated with the DTC and BSC using the fact that the attenuation coefficients for the DTC and BSC are different from each other [29], [30].
Recently, deep-learning-based underwater image enhancement methods have been proposed [31]- [34]. For training deep networks, it is typically challenging to obtain the test datasets of the pairs of the real underwater image and its corresponding ground truth restored image. To obtain training images, Li et al. [31] synthesized underwater images from clear images, and Li et al. [32] employed a generative adversarial network [35] to generate realistic underwater images. However, owing to the lack of sufficient reality in the training datasets, the performance of deep-learning-based underwater image enhancement methods is limited [34].
The majority of the existing underwater image restoration methods estimate the related parameters based on simplified imaging models and/or assumptions, and thus frequently fail to remove the blur and haze artifacts completely. In this article, we propose a novel underwater image restoration algorithm based on the complete UIFM. We first estimate the transmission maps based on the observation that the scene distance is inversely proportional to the geodesic color distance from the background light. Although the majority of existing methods consider the DTC and BSC only, we regard an underwater image as the superposition of the backscattering layer of the BSC and scene layers of the DTC and FSC. Using the estimated transmission map, we decompose an input underwater image into the backscattering layer and scene layer. We also approximate the point spread function (PSF) associated with the FSC to extract the DTC from the scene layer. Then, we restore the scene radiance with additional white balancing using the optimal parameters of the UIFM determined by minimizing a cost function composed VOLUME 8, 2020 of the sharpness, information loss, and dark background prior (DBP). The experimental results demonstrate that the proposed algorithm estimates the transmission maps more faithfully and restores the original scene radiance more reliably compared with existing state-of-the-art methods.
The remainder of this article is organized as follows. Section II describes the complete UIFM. Section III explains the proposed algorithm and Section IV presents the experimental results. Section V concludes the paper.

II. UNDERWATER IMAGE FORMATION MODEL
The complete UIFM is composed of three components, the DTC, FSC, and BSC [6].
where I c (x) and J c (x) denote the intensities of the c ∈ {red, green, blue} color channel at the pixel location x in an input underwater image and scene radiance image, respectively. The term J c (x)t c (x) is the DTC that represents the attenuated scene radiance by transmission where d(x) denotes the distance at x between a target scene and camera, and β c denotes the attenuation coefficient of the (1) is the FSC where k c (x) denotes the PSF defined at x given by R c and h c are the empirical coefficients of the c color channel related to the blur artifact, such that |R c | < |β c |. F −1 denotes the inverse Fourier transform and ω is the radial frequency. The term B c (1 − t c (x)) in (1) corresponds to the BSC where B c is the background light of the c color channel, which is the accumulated BSC from infinitely far to the camera. Note that B c is highly correlated with the ambient light [26] which depends on the water depth of the target scene. In this article, we assume that β c and B c are constant over the entire image for each color channel.

III. PROPOSED ALGORITHM
For a given input underwater image, we estimate the transmission map using the geodesic color distance from the background light. Then, we decompose the input image into the backscattering and scene layers based on the UIFM, where the DTC is extracted from the scene layer. We restore the scene radiance from the DTC and remove the color cast. The optimal parameters of the UIFM are determined by minimizing a cost function composed of the sharpness, information loss, and DBP.

A. TRANSMISSION ESTIMATION
Note that according to (1), I c (x) ≈ J c (x) when t c (x) approaches one; simultaneously, the scene distance d(x) approaches zero. This means that the original scene radiance for objects close to the camera is highly likely to be conveyed intact without severe haze. Conversely, as the scene distance becomes greater, it is mainly the background light that attenuates the scene radiance and eventually I c (x) ≈ B c when t c (x) ≈ 0 at the infinitely far regions. We estimate the transmission value at each pixel by deriving the relationship between the scene distance and contribution of the background light to the intensity of the input underwater image. Fig. 2(a) displays an underwater image and Fig. 2(b) indicates its scene distance map provided in [36], where pixels with invalid distance values are colored in blue. We first estimate the background light B = [B r , B g , B b ] using [20]. We partition the input image into superpixels using SLIC [37], and extract the far background regions as the set , representing top 10% superpixels with average colors most similar to the background light. When the ratios are excessively small, less than 10%, cannot represent the characteristics of the entire far background region. Conversely, with considerably greater ratios than 10%, certain superpixels of the foreground scene structures can be included in , which can cause an incorrect estimation of the far background regions. Then, we construct a graph, where the nodes correspond to the superpixels and edges are defined between all the pairs of adjacent superpixels. We assign a weight to each edge as the Euclidean distance of the colors between the two connected superpixels, where the color of the superpixel is computed as the average of the pixel colors in the superpixel. We define the geodesic color distance G(p) of each superpixel p from the background light as where z(p, q) is the geodesic distance between two superpixels of p and q along the shortest path of the connected superpixels, which yields the minimum value of the sum of edge weights among all the possible paths from q to p. Fig. 2(c) displays the geodesic color distance map of the input image in Fig. 2(a), where the superpixels in are depicted in red. As observed in Figs. 2(b) and (c), the scene distance is inversely proportional to the geodesic color distance from the background light. Fig. 2(d) plots the distribution of the pairs of G(p) and d(p), where d(p) is the average scene distance of the pixels in p. We notice that the data points are distributed along a red curve. Fig. 3 empirically plots the pairs of G(p) and d(p) for more images provided in [28], [36] capturing diverse underwater scene structures. We can also observe similar results of the inverse relationship between G(p) and d(p), which is modeled by where τ and δ are positive parameters. Fig. 3(b) displays the best fitted curves on the data distributions in red obtained by determining the optimal parameters of τ and δ in (5) using RANSAC [38]. Based on this observation, we estimate the transmission for each superpixel from the geodesic color distance from the background light. We assume that the scene distances at the pixels in a certain superpixel are identical to their average distance. At each superpixel p, we first obtain an initial transmission in the blue channel as a reference, by substituting (5) into (2), given bŷ where α = e −β b τ . Then we compute the transmission in the other color channels using the ratios of the attenuation coefficients [13], [26] aŝ where γ c = β c β b . Figs. 4(b) and (c) display the geodesic color distance map and initial transmission map in the blue channel for a given image in Fig. 4(a). We can observe that the background, which is far from the camera, has low transmission values, whereas the close scene structures to the camera such as the floor and rocks are assigned relatively high transmission values. However, the initial transmission map exhibits discontinuities at the boundaries of the superpixels. To obtain refined transmission maps while preserving the edges of the scene structures, we apply guided image filtering [39] on the three initial transmission maps using each channel of the input image as a guide image. The resulting refined transmission maps in the red, green, and blue channels are presented in Figs. 4(d), (e), and (f), respectively. The blocky artifacts in the initial transmission map are successfully alleviated in the refined transmission map. Note that we have an arbitrary input underwater image without knowing its scene distance map. Therefore, the optimal parameters of α, δ, and γ should be estimated to compute the transmission from the geodesic color distance. The parameter estimation will be explained in Section III-C.

B. SCENE RADIANCE RESTORATION
Based on the UIFM in (1), we interpret an underwater image as the superposition of the scene layer associated with the scene radiance J c (x), and the backscattering layer which covers the scene radiance. We first estimate the backscattering layer using the background light and transmission estimated in Section III-A, which is then subtracted from the input underwater image leaving the scene layer. Fig. 5 displays examples of layer decomposition. We can observe that the proposed algorithm can bleach the input underwater images leaving the scene layers by subtracting the backscattering layers effectively regardless of diverse background lights with distorted colors. To obtain the scene radiance, we estimate the PSF k c (x) in (3), and further decompose the scene layer of (8) into the DTC and FSC. Note that it is typically difficult to estimate the weight term of (e −R c d (x) − e −β c d(x) ) in (3) completely, and thus we regard it as constant for simplicity [7], [40] and suggest estimating k c (x) using the transmission. We rewrite k c (x) in terms of t c (x) as where η c = R c β c ∈ (0, 1) because |R c | < |β c |, and ν c = h c β c > 0. Large values of η c are associated with clean water and small values of η c with turbid water. Note that when η c = 1, the FSC can be ignored. as in the majority of the existing methods based on sUIFM [8]- [14]. To simplify the problem, we assume that the transmission is constant in each color channel, and that η c and ν c are constant across the color channels. Then, we obtain an approximated PSFk c (x) given byk wheret c denotes the average transmission value in c color channel.
To extract the DTC from the scene layer, we usek c (x) to (8) and take the Fourier transform and inverse Fourier transform in order. Then, we have We estimate the scene radiance aŝ where c is a parameter to avoid division by zero. We set c = ( b ) γ c following the relationship in (7) and b = ( ) γ m . We empirically determine = 0.1 and γ m = min(γ −1 r , γ −1 g , 1), because excessively low values of cause saturation, whereas overly high values of yield limited contrast enhancement. Fig. 6(b) displays examples of the clipped restored scene radiance images, J c (x) = max(Ĵ c (x), 0); however, we continue to have a certain amount of global color cast due to the color distortion in the ambient light. We remove this color cast from J c based on the Gray World Assumption [27] and estimate the white balanced scene radiance O c as where A c denotes the average of J c (x) in an image andĀ denotes the average of A c over the three color channels.
As indicated in Fig. 6(c), the greater part of the global color cast is effectively removed in the white balanced scene radiance images.

C. PARAMETER ESTIMATION
To obtain the transmission and restore the scene radiance, we require six parameters, α and δ in (6), γ r and γ g in (7), and η and ν in (11). Let us denote the set of the parameters as = [α, δ, γ r , γ g , η, ν]. The optimal parameters can vary according to the capturing environments of underwater images, e.g., the turbidity or water type. In this Section, we estimate the optimal set of parameters, * , to restore the scene radiance image with a minimum quality degradation including haze, blur, color distortion, and low contrast.

1) COST FUNCTION
We formulate a cost function to quantify the quality degradation of the restored scene radiance, given by The first term measures the average sharpness of the image, which is defined as where N is the number of pixels in the image and ∇ is the gradient operator.Ĵ c (x) is the intensity of the scene radiance at x in the c color channel when restored with the parameter set . Note that the original scene radiance is attenuated and blurred while traveling to the camera. Moreover, it loses information owing to the haze caused by the backscattering of the light. Therefore, it would be beneficial to increase the sharpness E sharp ( ) of the restored image. In addition, the restored scene radiance values could fall beyond the valid range of [0, 1] by (13), which could cause the information loss. Hence, we also employ an information loss cost On the other hand, the BSC can also increase the sharpness of an image, and therefore, the restored scene radiance obtained by combining (16) and (17) frequently retains the BSC, especially in the far background regions. To bleach the BSC reliably from the scene radiance image, we, in addition, propose a DBP: the scene radiance transmitted from infinitely far is completely absorbed by the water, and therefore, in an underwater image the far background region contains only the BSC and should be dark in the restored scene radiance. Specifically, we formulate this prior using the geodesic color distance from the background light because a large scene distance d(p) implies small geodesic color distance G(p) according to (5) and low scene radiance intensity J c (p) accordingly. In practice, we define a weight w(x) as where κ controls the effective area of the far background region; a higher value of κ is associated with a larger background area and vice versa. We empirically set κ = 5×10 −3 . Then, we employ the cost function for DBP given by whereJ (x) is the intensity of the restored scene radiance averaged over the three color channels,J (x) = 1 3 cĴ c (x). Note that the behavior of (19) encourages the far background regions to have dark intensities, whereas the close scene structures preserve their scene radiance information. Finally, we minimize the total cost function in (15) and obtain reliable transmission maps and faithful scene radiance images with less of the BSC in the far background regions accordingly. In this work, we empirically set λ 1 = 2 and λ 2 = 0.1 in (15). Higher values of λ 1 limit the performance of the contrast enhancement, whereas lower values of λ 1 tend to cause saturation due to over-enhanced contrast. Moreover, as we increase λ 2 , the estimated transmission values decrease in the far background regions causing color distortion and saturation. Conversely, as indicated in the ablation study, excessively low values of λ 2 retain the BSC in the far background regions.

2) OPTIMIZATION
We first constrain the valid ranges of the parameters in = [α, δ, γ r , γ g , η, ν]. Note that the parameters in are all nonnegative. We empirically observed that δ < 1 for the majority of the test images in [36]. We consider the measured values of γ r and γ g for Jerlov water types [13], [25] to set their valid ranges, and ignore water types with γ −1 r > 1, which rarely occur. Consequently, the valid ranges of the parameters are set as α ∈ Unfortunately, it is difficult to find the optimal solution to globally minimize the cost function E( ) in (15) owing to the high dimension of ∈ R 6 . Instead, we iteratively optimize partial sets of the parameters to search for the local minima of E( ) by employing the Nelder-Mead (NM) simplex search algorithm [41] and alternative optimization [42]. We first initialize by setting each parameter θ ∈ as a randomly selected value from N θ (µ, σ 2 ), the truncated normal distribution with mean µ and standard deviation σ , where the range of N θ (µ, σ 2 ) is same as the aforementioned valid range of θ. We set µ = 0.9 for γ −1 g , µ = 0.5 for the others, and σ = 0.1 for all the parameters.
Then, we partition into three groups, θ 1 = [α, δ] associated with the transmission estimation in the blue channel, θ 2 = [γ r , γ g ] associated with the transmission estimation in the red and green channels, and θ 3 = [η, ν] associated with the estimation of the PSFs. The three groups of parameters are alternatively optimized. At the s-th iteration, we find the optimal solution for θ s 1 while fixing θ s−1 2 and θ s−1 3 . Then, we find the optimal solution for θ s 2 while fixing θ s 1 and θ s−1 3 . Finally, we find the optimal solution for θ s 3 while fixing θ s 1 and θ s 2 . This process is repeated until s ≈ s−1 . The overall process is summarized in Algorithm 1. We set ζ = 0.1 and s max = 10.  s ← s + 1 Specifically, to solve each step of lines 4-6 in Algorithm 1, we employ the NM algorithm three times and use the best solution as θ s in terms of E( ) in (15). The NM algorithm requires three initial points to construct an initial simplex as θ ∈ R 2 . We suggest reusing θ s−1 as an initial point to find θ s to guarantee the convergence of the proposed algorithm, and randomly select the other points from N θ (µ, σ 2 ). Moreover, if any side of the initial simplex is less than 0.1, new points are randomly selected again to avoid an overly small simplex. Fig. 7 displays the restored white balanced scene radiance images with iteratively estimated sets of parameters. We can observe that the quality degradation factors of the underwater images, such as blur, haze, and color distortion, are alleviated gradually as the parameter estimation is repeated.

IV. EXPERIMENTAL RESULTS
We defined DB 1 as the set of underwater images with corresponding scene distance maps by collecting 56 images from [36] and 91 images from [28]. We also employed DB 2, a set of ordinary underwater images without scene distance maps, 80 images from [43] and 10 images from [44]. We excluded the images that did not contain far background regions. We applied the preprocessing scheme in [13] to the images in DB 1, which were then downsampled to 512 × 342 images to reduce the computational complexity.

A. EVALUATION OF THE PROPOSED ALGORITHM
We evaluated the performance of the proposed algorithm in Fig. 8, where the top four rows are the images from DB 1 [28], [36] and the others are from DB 2 [43], [44]. As indicated in Fig. 8(a), the input underwater images exhibited different quality degradations caused by the water, such as haze, blur, color distortion, and low contrast. The proposed algorithm reliably estimated the transmission maps in the blue channel in Fig. 8(b), where the far background regions were assigned low transmission values, whereas the near scene structures had high values. It is worth noting that although we derived the formula in (5) from the images in DB 1, we could also obtain reliable transmission maps of the images in DB 2. The proposed algorithm decomposed the input images into the backscattering and scene layers, as indicated in Figs. 8(c) and (d), respectively, where we can observe that the overall effect of the background light was successfully removed in the scene layers. In particular, the proposed algorithm darkened the far background regions, which have virtually no scene radiance information, by removing the BSC completely with the assistance of accurate transmission maps. Then, from the scene layers, we restored the scene radiance images in Fig. 8(e), where we can observe that the details of the underwater objects and scenes such as the rocks and coral reefs are clearly visible by successfully removing the haze and blur and improving the image contrast. Finally, the color cast was alleviated by white balancing as indicated in Fig. 8(f). The experimental results demonstrate that the proposed algorithm can remove the quality degradation from underwater images and restore the original scene radiance.
We also measured the running time of the proposed algorithm. We implemented and executed the proposed algorithm in MATLAB on an Intel i7-9700F @ 3.00 GHz CPU. The running time depended on the number of iterations required to estimate the optimal parameters in Algorithm 1; the average iteration numbers were 3.18 for DB 1 and 3.26 for DB 2, meaning that 2 was selected as optimal in the majority of cases. We can observe that the average running time for DB 1 was 50.9 seconds, where the images in DB 1 were downsampled to have a spatial resolution of 512 × 342. The images in DB 2 had diverse resolutions from 404 × 303 to 750 × 1000, and yielded a running time of 147.9 seconds on average.

B. ABLATION STUDY
We also tested the effect of the FSC and DBP in the proposed algorithm.

1) FSC
Whereas the majority of the conventional methods [10], [11], [13], [19], [20] are based on the sUIFM, which ignores the FSC in the UIFM, the proposed algorithm considers the FSC in (1). Fig. 9 compares the results of the proposed algorithm with and without FSC. In practice, we set η = 1 and skipped line 5 in Algorithm 1 to ignore the FSC. In Fig. 9(b), we can observe that the restoration results without FSC were frequently saturated, e.g., the first scene, or color distorted, e.g., the second scene; however, considering the FSC yielded the restored results with less saturation owing to the denominator in (12) being greater than one, and improved the color correction with more reliable transmission values, as indicated in Fig. 9(c). Moreover, the FSC further improved the performance of the haze and blur removal, e.g., the rock and floor remain hazy and blurred in Fig. 9(b), which are sharpened in Fig. 9(c). Consequently, the underwater image restoration based on the complete UIFM provided more faithful and detailed scene radiance compared with that of the sUIFM.
2) DBP Fig. 10 compares the restoration results of the proposed algorithm with and without DBP. We set λ 2 = 0 in the cost function in (15) to ignore the effect of DBP. When DBP was not considered, an amount of blue color remained in the BSC to increase the image contrast in the background regions via E sharp in (15). In such cases, the white balancing relatively amplified the red and green colors while suppressing the blue color on average, and hence caused a yellowish color distortion, especially on the close scene structures, e.g., the rocks and deck of the sunken ship as indicated in Fig. 10(b). Conversely, DBP encouraged a reliable estimation of the transmission maps and assigned low transmission values to the far background regions. Therefore, the scene radiance was restored more faithfully by removing the bluish colors from the background regions more completely, as indicated in Fig. 10(c).

C. COMPARISON WITH CONVENTIONAL METHODS
We compared the performance of the proposed algorithm with six state-of-the-art methods [12], [13], [19], [20], [31], [40]. We generated the results using the source codes provided by the authors with the default parameters.

1) PERFORMANCE OF TRANSMISSION ESTIMATION
We first compared the estimated transmission maps of the proposed algorithm with those of the three state-of-theart methods with available transmission maps: UDCP [12], IBLA [20], and NLP [13]. Fig. 11 displays the estimated transmission maps with the provided scene distance maps. From top to bottom, each test image was selected from the 'Nachsholim', 'Katzaa', 'Satil', 'Michmoret', and 'D5' categories of DB 1 [28], [36]. Because the assumption of DCP is not valid for the images in DB 1 [28], [36], UDCP failed to estimate reliable transmission maps in the majority of the cases. For example, the floors of the images in the 'Nachsholim' and 'Michmoret' categories were assigned lower transmission values than the rocks or the background owing to brighter colors, as shown in Fig. 11(c). IBLA uses VOLUME 8, 2020  the blurriness to estimate the transmission, and assigned similar transmission values to the homogeneous regions of the floor and background as indicated in the first and fourth rows in Fig. 11(d). Conversely, NLP, which uses non-local prior, provided relatively faithful transmission maps where the background regions had lower values than the close objects, e.g., in the first row in Fig. 11(e). However, it assigned high transmission values to the background as indicated in the second and fourth rows in Fig. 11(e), and even assigned higher transmission values to the far rocks than the close floor, which is not true, as indicated in the last row in Fig. 11(e). Compared with the existing methods, the proposed algorithm estimated reliable transmission maps in the majority of cases, as shown in Fig. 11(f), where the close objects were assigned higher transmission values than the far background regions. Moreover, the far background regions had virtually zero transmission values, except for the image in the 'Satil' category. However, note that all the existing methods also failed to estimate the correct transmission values for this challenging image with non-homogeneous background lights as indicated in the third row in Fig. 11.
Berman et al. [36] proposed a metric to measure the performance of the transmission estimation using the Pearson correlation coefficient ρ ∈ [−1, 1] between the ground truth scene distance map and negative logarithm of the estimated transmission map. Table 1 quantitatively evaluates the performance of the transmission estimation in terms of the correlation coefficients averaged over all images in the images of DB 1. UDCP and IBLA yielded relatively low values of ρ. NLP indicated a higher score than UDCP and IBLA. However, the proposed algorithm achieved the highest score. TABLE 1. The average Pearson correlation coefficient ρ evaluated on DB 1. The higher ρ implies that the negative logarithm of the estimated transmission map is highly correlated with the scene distance map. The best score is colored in red.

2) PERFORMANCE OF SCENE RADIANCE RESTORATION
In Fig. 12, we qualitatively compare the underwater image restoration results of the proposed algorithm with those of the seven state-of-the-art methods: UDCP [12], SDCP [40], MIL [19], HP [19], IBLA [20], NLP [13], and WCNN [31]. Note that [19] first restores the scene radiance based on the sUIFM with a minimum information loss, and then applies enhancement methods to the restored scene radiance. Thus, we compared both of the scene radiance restoration method as MIL, and the histogram prior-based enhancement method as HP. We also compared the results of NLP without applying a standard image processing pipeline. Because there are eight different sets of pre-trained parameters for WCNN [31], we generated all the results and selected the most visually pleasing result for each test image. UDCP and SDCP employ modified versions of DCP [21], which is not valid for typical underwater images, and thus have limited performance ability for removing the BSC in the background regions as indicated in Figs. 12(b) and (c). Moreover, they frequently provide saturated results, e.g., the floor in the first scene FIGURE 11. Comparison of the estimated transmission maps on DB 1 [28], [36]. (a) Input images. (b) The scene distance maps provided in [28], [36] where the pixels with invalid distances are colored in blue. The estimated transmission maps in the blue channel obtained by using (c) UDCP [12], (d) IBLA [20], (e) NLP [13], and (f) the proposed algorithm, respectively. and fish in the fifth scene. SDCP restored more sharpened results by considering the FSC, as indicated in the first scene in Fig. 12(c). MIL and HP retained a substantial amount of haze and blur in the resulting images, e.g., the floor in the first scene and rock in the last scene as indicated in Figs. 12(d) and (e). As shown in Fig. 12(f), IBLA also demonstrated limited performance in the complete removal of haze and the BSC, and furthermore, frequently failed to correct the color distortions owing to an incorrect estimation of the parameters γ r and γ g . NLP restores the scene radiance by selecting the best parameters of γ r and γ g among the measured values for 10 Jerlov water types in terms of the Gray World Assumption [27], and therefore, demonstrated acceptable color correction performance, such as the first two scenes in Fig. 12(g). However, it is possible that 10 measured values are not optimal for diverse underwater images as indicated in the third and fourth scenes. Moreover, NLP indicated limited performance of haze removal and contrast enhancement. WCNN uses synthetic underwater images for network training. Owing to the domain gap between the real underwater images and synthesized images, the results of WCNN remained hazy and blurry as shown in Fig. 12(h). Conversely, the proposed algorithm estimates reliable transmission maps using the geodesic color distance from the background light, and hence removed the BSC in the far background regions well as indicated in Fig. 12(i). Moreover, the proposed algorithm restores the scene radiance based TABLE 2. Evaluation of the quantitative quality of the restored underwater images in terms of UCIQE [45] and UIQM [46]. The best and the second best scores are colored in red and blue, respectively. on the UIFM considering the FSC, and thus removed the haze and blur effectively and restored more sharpened scene structures, e.g., the second and third scenes in Fig. 12(i). The proposed algorithm finds the optimal values of γ r and γ g by minimizing the cost function of (15), and alleviates the color distortions well. In particular, as compared in the third scene of Fig. 12, UDCP, SDCP, and MIL failed to remove the color cast and thus resulted in bluish color charts in the restored images. Moreover, the results of HP were overly saturated, and the results of IBLA and NLP were actually greenish FIGURE 12. Qualitative comparison of the restored underwater images. The top four scenes are from DB 1 [28], [36], the next two scenes are from DB 2 [43]. (a) Input images. The restored scene radiance images obtained by using (b) UDCP [12], (c) SDCP [40], (d) MIL [19], (e) HP [19], (f) IBLA [20], (g) NLP [13], (h) WCNN [31], and (i) the proposed algorithm, respectively.
owing to the incorrect estimation of γ 's. Conversely, the proposed algorithm successfully reconstructed the color chart to become white and sharpened. In addition, as shown in the fourth scene of Fig. 12, the restored images of HP and IBLA suffered from saturation and color distortion, respectively, and those of the other existing methods continued to exhibit haze artifacts. The proposed algorithm removed the color cast successfully and restored clear scene structures faithfully.
Next, we evaluated the quantitative performance of the proposed algorithm in terms of two measures: underwater color image quality evaluation (UCIQE) [45] and underwater image quality measure (UIQM) [46]. Table 2 compares the scores of the two measures according to the two test datasets. A higher value implies a superior image quality. For WCNN, we generated eight results for each input image and selected the best score. Note that these are non-referencebased metrics designed to measure the quality in terms of the colorfulness, sharpness, and contrast. Therefore, it is possible they do not correctly evaluate the performance of the image restoration compared to the ground truth. Based on Table 2, HP achieved the best scores in terms of both UCIQE and UIQM, as HP is mainly designed to increase the image contrast. However, as observed in the qualitative results of Fig. 12, HP frequently resulted in overly saturated results due to over-enhanced image contrast. Despite the inadequacy of the quantitative measures in this work, the proposed algorithm yielded the second best UCIQE and UIQM scores for DB 1, and provided comparably high scores for DB 2.

V. CONCLUSION
In this article, we proposed a novel underwater image restoration algorithm based on the complete UIFM. We estimated the transmission with the observation that the scene distance is inversely proportional to the geodesic color distance from the background light. We decomposed an input underwater image into scene and backscattering layers, and restored the scene radiance with the optimal parameters of the UIFM, which were obtained by minimizing a cost function reflecting the sharpness, information loss, and DBP. The experimental results demonstrated that the proposed algorithm estimated the transmission map more accurately, and therefore achieved considerably superior scene radiance restoration performance compared with the existing state-of-the-art methods.