An In-Depth Survey of Underwater Image Enhancement and Restoration

Images taken under water usually suffer from the problems of quality degradation, such as low contrast, blurring details, color deviations, non-uniform illumination, etc. As an important problem in image processing and computer vision, the restoration and enhancement of underwater image are necessary for numerous practical applications. Over the last few decades, underwater image restoration and enhancement have been attracting an increasing amount of research effort. However, a comprehensive and in-depth survey of related achievements and improvements is still missing, especially the survey of underwater image dataset which is a key issue in underwater image processing and intelligent application. In this exposition, we first summarize more than 120 studies about the latest progress in underwater image restoration and enhancement, including the techniques, datasets, available codes, and evaluation metrics. We analyze the contributions and limitations of existing methods to facilitate the comprehensive understanding of underwater image restoration and enhancement. Furthermore, we provide detailed objective evaluations and analysis of the representative methods on five types of underwater scenarios, which verifies the applicability of these methods in different underwater conditions. Finally, we discuss the potential challenges and open issues of underwater image restoration and enhancement and suggest possible research directions in the future.


I. INTRODUCTION
Underwater optical imaging systems mainly include an optical camera, or adopt techniques such as polarization, stereo/panoramic, and spectral imaging [1]- [4].However, each of techniques other than optical cameras has its limitations, such as a narrow field of view, limited depth, complex and professional operation, etc.When light propagates through the water, the absorption and scattering determined by the internal optical property (IOP) of the water affect the process of underwater imaging [5].Specifically, forward The associate editor coordinating the review of this article and approving it for publication was Mehul S. Raval.
scattering occurs when the light reaches the receiver after being reflected from the target objects.The forward scattering makes the point light source diffuse into a blur circle, which results in blurred images.The backscattering reduces the contrast and produces foggy veiling in an underwater image.Besides, the dissolved organic matter and small floating particles which are called 'sea snow', whose concentration and species vary greatly, also affect underwater image quality [6].With the depth increasing in water, the colors of light disappear depending on their wavelengths.Although artificial lighting can be used to increase the visible distance, it produces bright spot in the image surrounded by a dark area and makes the scattering caused by suspended matters more serious.In addition, the inherent noise of underwater imaging system also is a significant factor which affects the quality of underwater image.Therefore, the optical images captured from water need further restoration or enhancement processing to improve their visual quality.
According to underwater imaging model was applied or not, the related works can be divided into two categories, underwater image enhancement methods and underwater image restoration methods.Usually, subjective and objective underwater image quality evaluations were employed to evaluate the performance of different methods.With the development of artificial intelligence technologies, many remarkable achievements have been made in underwater image restoration and enhancement.Some topics that are related to underwater image restoration and enhancement have been studied in [7], [8].However, an in-depth exploration of underwater restoration and enhancement methods, image quality evaluation techniques and related datasets is still missed.The contributions of this paper are listed as follows: (i) we refer to more than 120 related studies and summarize existing techniques, datasets, and evaluation metrics, which aims to help researchers to understand the development of this research area; (ii) we conduct detailed objective evaluations and comprehensive performance analysis for the representative methods under classical five types of underwater scenes, which can guide the selection of the most appropriate method for practical cases; (iii) we summarize the datasets widely used in the representative researches, which is the most concerned issue in underwater image artificial intelligent exploitation; and (iv) we look into several open issues of underwater image restoration and enhancement, which sheds light on potential research directions in future.
The rest of this paper is organized as follows.Section II surveys the recent underwater image restoration and enhancement methods.Underwater image quality evaluations and datasets, followed by experimental results on five groups of underwater images are presented in Section III.Section IV discusses the open issues in underwater image processing.The conclusion of this paper is given in Section V.

II. SURVEY OF STATE OF THE ART
In addition to using the simulated images for testing [9]- [11], most of the underwater image processing methods focued on the real optical underwater images to improve their clarity, contrast and genuine color.In this section, we review the related works of two categories, underwater image restoration and underwater image enhancement.

A. UNDERWATER IMAGE RESTORATION METHODS
In general, underwater image restoration methods can be further categorized into four groups depending on the degradation models they adopted: (1)

1) PSF ESTIMATION BASED METHODS
In addition to measuring the PSF and modulation transfer function (MTF) of seawater in the laboratory [12], Hou et al. [13]- [15] regarded the imaging process in seawater as a linear system.They incorporated the optical properties of water into the traditional image restoration methods.The absorption, attenuation, particle distribution and volume scattering functions were measured by the specific instruments.At the same time, the model parameters were estimated by wavelet decomposition and power spectrum ratio after denoising process.Grosso [16], [17] and Voss and Chapin [18] also measured the PSF by the specific instruments.However, the instruments they used were complex and expensive.Moreover, these methods are difficult to meet the requirements of real-time processing.
Cho and Kim [19] measured the depth of the scene by Doppler velocimeter.The illumination of artificial light was estimated based on the model of LED optical transmission [20].At last, Cho and Kim restored underwater images by denoising, defogging, and deconvoluting by using the PSF based on the generalized Gaussian distribution, and then stretched the contrast by the contrast-limited adaptive histogram equalization (CLAHE).
Beyond these, the deep-sea underwater camera [21], stereo cameras [22] and laser range-gated underwater imaging [23], [24] were taken into account to restore the underwater images.

2) JAFFE-MCGLAMERY MODEL BASED METHODS
Except for some models which aimed to study the influence of illumination beam characteristics on the imaging process [25], the Jaffe-McGlamery underwater imaging model [26]- [28] was widely used in underwater image restoration methods, in which the light E T received by the camera was decomposed into three parts: the light reflected directly from an object E d , the forward scattered portion E f (small-angle light reflected from a target) and the backscattered light E b (non-target reflected light), as given by Eq. ( 1) and shown in Fig. 1.
Trucco and Olmos [29] proposed a self-calibrated filter based on a simplified Jaffe-McGlamery model.The filter was VOLUME 7, 2019 123639 designed based on two assumptions: (1) illumination (direct sunlight in shallow water) was uniform, and (2) forward scattering was the main attenuation component while the backscattering and the direct component were ignored.For an underwater image, the optical parameters were estimated by optimizing a global contrast evaluation function (minimum blur).They qualitatively and quantitatively evaluated the effect of the restoration for image classification (whether there were artificial targets) [30], [31].Wang and Wu [32] focused on the backscattering in the Jaffe-McGlamery imaging model, and on the basis of the dark channel prior (DCP) [33].This method assumed that there was a region with high contrast in the image which was not affected by backscattering.Based on this assumption, the model parameters were estimated.Besides the limitations of the dark channel method, there was no objective image evaluation, and the restoration results were over-saturated.Akkaynak and Treibitz [34] presented a revised underwater imaging model based on Jaffe-McGlamery model, in which they treated the direct and backscattering coefficients separately.By utilizing the measured depth of the field, they estimated the attenuation coefficient and restored color image spatially.
The deconvolution method is strict but difficult to implement because the model parameters in the scene are unknown and change with time and space in most cases.Moreover, the execution time of the blind restoration is relatively long, which is not suitable for the real-time applications.

3) TURBULENCE DEGRADATION MODEL BASED METHODS
Turbulence leads to a random change in the refractive index of the atmosphere, which is similar to the light propagating in water.Hufnagel and Stanley [35] proposed an image degradation model H based on the physical properties of atmospheric turbulence, which can be expressed in the frequency domain (u, v) as: where k indicates the extent of turbulence.Underwater image restoration based on the atmospheric turbulence image degradation model represents a process of estimating the parameter k.By combining the atmospheric turbulence image degradation model with the image quality evaluation function, the adaptive underwater image restoration can be realized.For instance, Zhang et al. [36] combined Wiener filter with the image quality evaluation to estimate model parameters and restored images based on the turbulence degradation model.Yang and Wei [37] proposed an adaptive underwater restoration scheme based on the turbulence model, wherein the weighted contrast average grads (WCAG) was used to evaluate the quality of an underwater image.

4) IMAGE DEHAZING BASED RESTORATION METHODS
In this subsection, we group the underwater dehazing methods in two subgroups: (1) classical DCP based restoration methods, and (2) learning based DCP underwater image restoration methods.These methods are listed in Table 1, and discussed at the end of this section.

a: CLASSICAL DCP BASED IMAGE RESTORATION METHODS
In the Jaffe-McGlamery underwater imaging model, for a degraded underwater image I c (x) , c = {R, G, B}, E d is given by: where j c (x) is the undistored underwater image, and d(x) is the distance between the observer and the object, p λ denotes the sum of the absorption coefficient a λ and scattering coefficient b λ in an underwater environment, both of which are related to the wavelength λ: The exponential term e −p λ d(x) is referred to as a transmission map t c (x): Further, E b is given by: where A c ∞ denotes the background or backscattering light of a color channel c.Generally, forward scattering E f is related to a small part of the image degradation process, so a simplified underwater optical model which is used in most existing underwater restoration methods can be expressed as: By substituting the original image I c (x), A c ∞ and t c (x) into (7), the restored underwater image J c (x) can be expressed as: He et al. [33] proposed the dark channel hypothesis and pointed out that an increase of the dark channel brightness was due to the fog.I Dark is the dark channel of I c (x) obtained from the local minimization of R, G and B channels, which is expressed by: where (x) is the neighborhood centered at a pixel x.According to the DCP, at least one color channel in J c (x) is assumed to have zero pixel value, thus: Then, I Dark can be obtained by applying Eqs. ( 10) and (9) on Eq. ( 7): Usually, Ãc ∞ represents the estimation of A c ∞ , and is defined as the maximum value of the dark channel, and the  global background light is used as a denominator to estimate the transmission map tc (x), which is respectively given by: In recent years, the underwater image restoration based on DCP has received extensive attention [34], [38]- [58].
An assumption that the red attenuation is the fastest, which is basically correct in open water, has been used to compute the dark channel image in several DCP based underwater image restorations.Carlevaris-Bianco et al. [38] first calculated the maximum difference between the blue-green channel and red channel, and estimated the transmission map by adjusting the difference until the maximum difference was 1.The minimum value of the transmission map was used as a background light.At last, the restored image was estimated by maximizing the posterior probability.Chiang and Chen [39] defined the transmission map as a residual energy ratio of the original image to the camera after reflection.The artificial light source was estimated by comparing the average brightness difference between the foreground and background.Galdran et al. [40] considered the red channel as an underwater prior.The background light was estimated by the maximum value of the red channel.P. Drews, Jr., et al. [41] assumed that the red channel attenuated the fastest, so it could not provide information related to the field depth.Therefore, an underwater dark channel (UDCP) prior was proposed, where the dark channel image was calculated by using the minimum of G and B channels, and the background light was estimated by the maximum value of the obtained dark channel image.
The inhomogeneous color projection caused by the absorption of light through water often makes the dark channel prior fail to estimate the transmission map accurately.In addition, an underwater environment is usually characterized by a little or inadequate light.A dark scene point will still be dark after imaging and will be erroneously estimated to be closer to the camera in the application of the DCP.In recent works, the field depth and fuzzy image have been utilized to improve the transmission map estimation [42], [45]- [46], [49], [51], and the color correction has been widely combined to compensate for a non-uniform color projection caused by absorption [42], [45]- [47], [49]- [53], [56]- [58].
Background light, can also be defined as a flat area [47], [50] or blurriest region [44], [56].Li et al. [47] computed the regional variance and the corresponding fuzzy graph of the local image by iterative quadtree decomposition.The weighted combination of the maximum and minimum values of a set consisted of the mean of 0.1% blurred image elements, pixels in the minimum variance region, and the mean of pixels in the fuzziest region was adopted as a background light.Furthermore, a combination based on different underwater images and light conditions was applied to estimate the depth of field.Emberton et al. [44] proposed a hierarchical method to find the blurriest region in an underwater image and estimate the background light.However, this method became unreliable when the color of the underwater target was close to that of the blurred area.Based on the hierarchic VOLUME 7, 2019 123641 technique, Emberton et al. [56] further divided underwater images into three categories: bluish, blue-greenish and greenish, and different white balance procedure was utilized for each category before the DCP based restoration.However, when the theoretical maximum value of the background light was used as the denominator to estimate the transmission map, the phenomenon of over-saturation occurred, which resulted in artifacts in the background region [59].
As the imaging effects also depend on object distance, the degradations are local and cannot be corrected by global operations.The related research has shown that the light rays traveling through the underwater environment encountered beam-particle interaction with different random times [60].A single background light value over the entire image failed to explain the real interactions between light rays and particles in the water medium, and the enhanced underwater image experienced phenomena such as poor local clarity and local over-saturation.On the other hand, since the 3D space had to be sliced into planes to calculate the backscattering irradiance of every small plane in different directions and distances relative to the camera, the background light was regarded as a superposition of many point-sources of the light in the space, which produced a non-uniform image intensity.Therefore, multiple background values were utilized in [48], [58].Ancuti et al. [48] estimated the background light by the local maximum value of the dark channel.Yang et al. [58] explored the statistical priori of offshore images to compute the dark channel.In the proposed method, Retinex reflection light decomposition was applied to the dark channel and the backscattering light was obtained by local Gaussian lowpass filtering of R, G and B components of a raw image respectively, and color correction was added to the recovered underwater image to compensate a possible color deviation further.
Beyond these, Cho et al. [46] trained an incremental Gaussian processing (iGP) [61] with the Flickr dataset [62], assuming the local sparse depth data had been known; they estimated the transmission map in an online manner [63].After that, they used the brightest pixel in the depth of field as the global background light.In the restoration process, the background light was compensated by a color correction in the CIELab space.
In addition to color stretching after defogging, more and more methods incorporated color compensations [50], [52]- [53], [56].Ancuti et al. [52], [53] combined a color transfer algorithm with DCP based underwater image defogging.Li et al. [50] applied the classical DCP to a color corrected underwater image.They also assumed a global uniform background light.After the Gaussian filtering in the CIELab space, they located a highlight and low-gradient flat region in L component with the size larger than 5% of the image as a candidate region.For bluish or greenish underwater images, the corresponding color of pixel with the largest blue-red or green-red difference of the brightest 0.1% pixels in the candidate region was selected as a final global background light.A total of thirteen features, including the dark channel, local maximum contrast, local maximum saturation, and hue difference, were extracted from the simulated blurred images in the transmission map estimation process.Transmission map of the image block was acquired by the random forest model [64].

b: LEARNING BASED DCP UNDERWATER IMAGE RESTORATION
The majority of the existing learning-based studies on the DCP underwater restoration focuses on the supervised scenario [50].However, in some of these methods, the unsupervised techniques were utilized.In [54], [55], the authors clustered all the colors in natural image into 500 categories according to the statistical distribution of the color image.Each pixel in a color image was represented by a clustering center.Color pixels exhibited a line segment in clustering space according to the distance to the camera.The attenuation curve was obtained by the k-dimensional (KD) tree clustering with logarithmic of the RGB value.The background light was estimated by selecting the pixel value with the largest difference between R, G and B channels in the image blocks whose total variation was less than the predefined threshold.At the same time, saturation constraint was applied to adjust the transmission map, but the restored image was still dark and over-saturated.
Convolutional Neural Networks (CNNs) [65] have also been applied to underwater image restoration.Ding et al. [51] used a CNN to estimate the depth of a corrected underwater image with white balance, so as to estimate the transmission map.In the training process, the Make3D dataset [66] was adopted, and the mean color value of the corrected image was used as the global background light.Hou et al. [57] proposed an underwater residual CNN (URCNN) model by modifying the VGG network [67] to learn the transmission map.In the residual architecture they designed, a global background light was selected from blue and green channels as same with the strategy applied in [43].To train the proposed URCNN, they synthesized an underwater image dataset consisting 1000 images from the NYU Depth dataset [68] with realistic depths of object and the corresponding clean images.The illuminance compensation and color correction were also performed on the output of the URCNN to get the final image.

c: SUMMARY OF UNDERWATER IMAGE RESTORATION METHODS BASED ON DCP
The performance of underwater defogging based on the DCP can be affected by the background light and transmission map estimations, and the combined color correction methods.The estimations of background light and transmission map adopted in the literatures mentioned in this section are summarized in Appendix A. Fig. 2 and Fig. 3 illustrate the background light estimations and transmission maps of three underwater images obtained by different methods, respectively.
To summarize, different IOPs in water make the dark channel priors assumed by various algorithms unsuitable for other   [38], the output of the method of Galdran et al. [40], the output of the method of Peng and Cosman [49], the output of the method of Li et al. [47], and the output of the method of Yang et al. [58].

FIGURE 3.
The transmission map estimation results obtained by different restoration methods.From left to right: Input image, the output of the method of Carlevaris-Bianco et al. [38], the output of the method of Galdran et al. [40], the output of the method of Peng and Cosman [49], the output of the method of Li et al. [47], and the output of the method of Yang et al. [58].

B. UNDERWATER IMAGE ENHANCEMENT METHODS
Underwater image enhancement methods extract image information without any prior knowledge about the environment.Therefore, these methods are more general than image restoration methods.Various underwater enhancements are included in underwater image processing and analysis assignments, which are mainly inherited from the methods applied to natural images [69]- [71].In this section, we review underwater image enhancement methods according to the aspects they focus on, such as noise removal, contrast stretch, combined improvement with multi-information and deep learning.Table 2 lists all the methods.

1) FILTERING BASED METHODS
Arnold-Bos et al. [72] proposed a pre-processing framework for the luminance component of an underwater image.They analyzed the possible range of noise in an underwater image by using a combination of deconvolution and enhancement methods.The plural Log-Gabor wavelet denoising was used to suppress the remaining sensor noise, suspended particle noise and various quantization errors.This adaptive smoothing filter improved the edge detection effect.Besides, the proximity of the histogram distribution for enhanced underwater image to the exponential form was analyzed, but no quantitative comparison was provided.Bazeille [73] proposed a method consisting of multiple filtering steps to improve the non-uniform illumination, suppress noise, enhance contrast, and correct color of an underwater image.Jia and Ge [74] proposed a nonsubsampled contourlet transform (NSCT) based adaptive total variation (ATV) for underwater image denoising.Then, they used the partial differential equation (PDE) to eliminate noise and reconstructed the frequency components.The peak signal to noise ratio (PSNR) and sharpness were used to evaluate the quality of the enhanced underwater images, but there was no comparative evaluation of image quality between this and the other related methods.

2) COLOR CORRECTION BASED METHODS
Chambah et al. [75] applied the automatic color equalization (ACE) on RGB channels separately, and weighted the outputs of three channels to enhance the accuracy of fish recognition in the video taken by the remote-control camera of the aquarium.The internal parameters of the ACE algorithm were appropriately adjusted.Ghani and Isa [76], [77] proposed a series of color correction schemes based on the Rayleigh stretching.According to the characteristics of Rayleigh distribution, the blue histogram of an underwater image in RGB space was stretched to a low grayscale, the red histogram was stretched to a high grayscale, and the saturation and value components of the underwater image in HSV space were also stretched.Torres-Méndez and Dudek [78] treated an underwater image as a Markov random field (MRF).The nodes visible in the random field represented the color values of a degraded underwater image, while the hidden nodes represented the true color values.They described the structural relationship between pixels and their surrounding neighborhoods by learning the corrected colors of sample pixels.The difference in CIELab space of the pixels was used as a cost function, and a belief propagation (BP) algorithm was used to estimate the true color of each pixel.The illumination source was used to obtain the ''ground-truth'' image.However, it is difficult to obtain the corrected and pre-processed underwater image blocks to construct the training set, so this method only enabled the color correction of the scenes included in the training set.Iqbal et al. [79] proposed an image enhancement method using an integrated color model for marine environment.Their method is based on a series of sliding stretching, such as contrast stretching in RGB space and saturation and brightness stretching in the HSI space.However, there was no quantitative analysis of the quality improvement.
Petit et al. [80] proposed an underwater image color correction method based on the optical attenuation inversion.In this method, the geometric rotation of the quaternion space was used to assign corresponding pixels of the background region to gray or low saturation color, while keep the target unchanged.
According to Retinex theory [88]- [90], the object color perceived in human eyes is closely related to the reflection characteristics of the object surface, but has a weak relationship with the object illumination characteristics.Fu et al. [82] proposed a variational Retinex model, wherein the CIELab spatial luminance component of the color-corrected underwater image was decomposed by the linear domain variational Retinex through 4-6 iterations.In [84], instead of Gaussian filter, bilateral and triangular filters were utilized on L, a and b components, respectively, and then fused according to the ratio of the values in RGB space, which solved the edge halo problem of classic Retinex model and reduced the color distortion to a certain extent.However, the effect of defogging and contrast enhancement for turbid water was not achieved.

3) IMAGE FUSION BASED METHODS
Based on an observation that various techniques contribute differently to image quality improvements, the fusion strategy was considered gradually.Ancuti et al. [81] proposed a fusion-based underwater image enhancement method, wherein the outputs of white balance color correction and bilateral filtering were weighted with the result of histogram equalization.Four fusion weights, including Gaussian contrast, local contrast, saliency and sensitometry, were computed to obtain a pixel-level fusion output.Moreover, they improved the white balance processing under the premise that the red channel attenuation was the fastest [83].Experimental results showed that this method could improve the exposure of dark area and the global contrast, and enhance image edge details.However, for different underwater environments, the weighted coefficients in the fusion process are difficult to determine.

4) DEEP LEARNING BASED METHODS
Basically, underwater image enhancement based on deep learning networks is limited by the requirement for a large number of label images which are difficult to collect in practice.Table 3 shows a list of deep learning based enhancement models developed for underwater images.A set of color-corrected underwater images [91] was used as training data in [85], wherein the authors constructed an underwater image enhancement model based on a CNN.In the training process, 55 features were used, and fitted to a 3D enhanced underwater image in the final step.To simulate the attenuation caused by the water body, a Water-GAN network was proposed for underwater image color correction [86].Similar with the normal generative adversarial networks (GANs) [92], two training sets were input to the WaterGAN, one of which consisted of natural images and the corresponding depth maps in the air, and the other one consisted of the underwater images taken in the laboratory and simulated underwater images obtained by the Jaffe-McGlamery model.In the color correction network, two improved end-to-end convolution SegNet [93] networks were used to estimate depth map and correct the color by using the estimated depth map.Three modules in the WaterGAN generator simulated the characteristics of underwater imaging; namely, G-I simulated the attenuation, G-II simulated the light scattering, and G-III simulated the halo effect.The discriminator [94] in the WaterGAN was designed based on CNN to classify the real and simulated underwater images.Although the simulated underwater image generated by the WaterGan network simulated the depth-dependent color and brightness attenuations in underwater imaging under certain conditions (depth 1-2 meters, fixed light source, water body, etc.), it could not represent the degradation associated with the imaging system, light source and seasonal water properties, such as sea snow noise, contrast reduction, and foggy blurring caused by complex scattering.
Li et al. [87] proposed a weakly supervised color migration model inspired by cycle-consistent adversarial networks (CycleGAN) [95] to correct the color distortion of deepsea underwater images.A forward mapping and a backward mapping functions between an underwater image and the air image, and the associated adversarial discriminators were included in this model.Several distortion functions were adopted in the forward and backward generators, including the adversarial losses LossGAN, periodic continuity Loss-Cyc, and structural similarity LossSSIM.The content and details of the underwater image were unchanged, although the color was corrected.

III. UNDERWATER IMAGE QUALITY EVALUATION AND DATASETS A. UNDERWATER IMAGE QUALITY EVALUTION
Image quality assessment (IQA) plays a very important role in the adaptive optimization design of an optical VOLUME 7, 2019 123645 imaging system, image transmission, image enhancement and restoration, image retrieval and classification [96].Objective image quality evaluation (IQE) methods can be classified by whether a reference image exists or not.For underwater images where a reference image cannot be obtained, a no-reference image quality metric is needed to measure the perceptual image quality.The traditional objective evaluation methods evaluate the distortion (such as Gaussian noise) of an image taken in air, rather than the authentic mixed degradation caused by water body, so they often fail to evaluate the quality of an underwater image.
Several quantitative metrics have been used to evaluate enhancement and restoration performance for grayscale underwater images.For instance, Schechner and Karpel [97] applied global contrast as a measure of underwater grayscale image quality.Hou et al. [14] measured the quality of a restored image by a metric based on the weighted gray scale angle (WGSA) for scattering blurred underwater images.Arnold-Bos et al. [10] defined a robustness index to measure the closeness of the grayscale histogram to the exponential distribution.This index was also applied by Bazeille et al. [73].Arredondo and Lebart [11] proposed a methodology to assess the robustness of underwater image noise removing quantitatively.The true motion of a sequence of the underwater video was supposed to be known, and the angular deviation between the estimated velocity and the actual one was measured.
As for underwater color images, two prominent noreference underwater image quality evaluation metrics were proposed [98], [99].Panetta et al. [98] proposed an underwater image quality measure (UIQM) method, in which underwater image colorfulness measurement (UICM), underwater image sharpness measurement (UISM) and underwater image contrast measurement (UIConM) were combined to evaluate the underwater image quality.The choice of weighted coefficients depends on the application purpose.For instance, when evaluating the correction result of the color deviation of an underwater image, a larger weight value of UICM should be allocated.The training data set used in [98] contained 30 randomly selected underwater images captured with different devices and under a different water depth.The mean opinion scores (MOS) of the tested underwater images were gathered from 10 researchers on image processing.The UIQM was adopted in some enhancement/restoration methods [49], [55], [84].Yang and Sowmya [99] presented an underwater color image quality evaluation (UCIQE) metric to quantify the non-uniform color cast, blurring and noise in the underwater engineering and monitor images, and this metric has been widely applied in underwater enhancements [47], [49], [50], [84].Forty-four test images and 12 observers who were required to rate the tested images on a 5-level scale were used to obtain the MOS.Both of the two metrics were designed by combining the image quality components in a linear manner, training the weights for better evaluation.
Underwater images are all dominated by the integrated degradation, including chroma decreasing, low contrast, nonuniform illumination, blurring, non-uniform color casting, and noise from complicated factors.The mixed distortions manifested in underwater image make it difficult to construct a universal image quality metric that can be applicable to all types of underwater environments.An inaccurate score was obtained for an underwater image with dark area, oversaturation and non-uniform brightness by using the existed underwater image quality metric as analyzed in Section C.

B. UNDERWATER IMAGE DATASET
Underwater image datasets are significant in the development of underwater image processing technology.This section summarizes the underwater image datasets, which were used by scholars in the underwater image restoration and enhancement processes, as listed in Table 4. Examples of the images of these datasets are shown in Fig. 4.However, there is no relatively complete underwater image dataset due to difficulty in collecting underwater images.The current underwater image datasets face a series of problems, such as single target object, little category and imperfect labeling information.These problems severely restrict the development of intelligent underwater image processing technology.

C. EVALUTION RESULTS AND ANALYSIS
In this section, several typical methods for underwater image restoration and enhancement described in this paper were tested to compare their subjective and objective performances and operating time for various underwater images.The experimental underwater images were divided into 5 groups, including blueish, greenish, and yellowish underwater images, offshore (whitish), and deep-sea underwater images.We compared several underwater image dehazing methods: the DCP method proposed by He et al. [33], the method of Galdran et al. [40], and methods combining the DCP with color correction: the method of Yang et al. [58], the method of Peng et al. [49], and the method of Li et al. [50].And the tested color enhancement methods included the ACE [122] method, the method proposed by Iqbal et al. [79],   [108] and the OUC-VISION underwater image dataset [110].(b) Images from the underwater Photography-fish database [111] and Underwater Rock Image Database [86], [109].(c) Images from the Port Royal Underwater Image Database [86], [109] and the HabCam underwater image dataset [112]- [114].(d) Images from the MOUSS underwater image dataset [113], [114] and the AFSC underwater image dataset [113], [114].(e) Images from MBARI underwater image dataset [113]- [115] and NWFSC underwater image dataset [113], [114].(f) Images from RUIE underwater image dataset [116], [117] and RGBD underwater image dataset [34], [118], [120].

1) SUBJECTIVE INSPECTION
The experimental results are shown in Figs.5-9.It can be seen that the outputs of the method of Galdran et al. [40], the method of Peng and Cosman [49], the ACE method [122], the method of Li et al. [50], the method of Yang et al. [58] and the method of Fu et al. [82] recovered color visually to a certain extent for all the five groups of underwater images, among which the ACE method [122], the method of Li et al. [50], the method of Yang et al. [58], and the method of Fu et al. [82] had better applicability.However, blurring of the dark regions and color artifacts existed in the results produced by the method of Li et al. [50], as shown in Figs.5-6(e), the third image in Fig. 7(e), and the first  image in Fig. 9(e).The method of Fu et al. [82] improved the color saturation but produced blurred details in the output images, as shown in Figs.5-9(i).The other DCP based   More specifically, the adoption of the red channel prior in the method of Galdran et al. [40] had a negative effect on the image color restoration under bluish water, because it greened the yellow target in the underwater image due to the compensation of red channel, and produced more blue chroma for the yellowish underwater images, as shown in Figs.5-6.The restoration methods proposed by Peng and Cosman [49] and Li et al. [50] reduced the contrast of dark areas, as shown in the first images in Figs.5-6 and 8(d)-(e).Besides, the application of the color migration in the method of Li et al. [87] was prone to incur color spots in the enhancement results, as shown in Figs.5-7(j).The enhanced images by the ACE method [122] exhibited color deviation for the offshore images, which contained red and green target, as shown in the second image in Fig. 8(g).In general, the method proposed by Yang et al. [58], which was based on the Retinex composition on dark channel and local background light estimation, had a better color restoration effect for all the kinds of the underwater images: it improved the contrast of dark regions, and clarified the details in the underwater images significantly.

2) OBJECTIVE EVALUATION
The restoration results were evaluated by the PCQI, UIQM, and UCIQE metrics, since these metrics were widely used to qualify the comprehensive performance of underwater images.The PCQI was proposed to compare the difference VOLUME 7, 2019 123649 TABLE 5. Image quality evaluations for blueish underwater images in Fig. 5.
between the original and the enhanced grayscale images.A value of 1 represented no difference between the processed image and the original image.The values less or greater than 1 indicated a change, but the change did not necessarily denote an improvement on image quality.The higher the values of UCIQE and UIQM of an underwater image were, the better the image quality was.The values of the three metrics of the five groups of underwater images are listed in Tables 5-9.
In Table 5-9, the PCQI values were close to 1, which indicated that differences between the processed images and original images were less obviously because no color information evaluation was included in the PCQI.The minimum values in Tables 5-9 corresponded to the output images whose overall brightness changed greatly, such as images shown in Figs. 5 and 7(e), Figs. 6 and 8(j), and Fig. 9(g).For the images presented in Figs.6-8(d), and Fig. 9(j), the output images contained very dark areas, which induced abnormally high global contrast, average saturation, and then induced high UCIQE values as listed in Tables 5-9.In Tables 5-9, it also can be seen that the UIQM values of images obtained by the method of Li et al. [87] were influenced by the color deviations in the enhanced images, which was represented as higher chroma variance and local contrast.The Pearson's linear correlation coefficient (PLCC), Spearma's rank ordered correlation coefficient (SROCC), and root mean square error (RMSE) of the MOS and the UCIQE and UIQM values of the images presented in Figs.5-9 are shown in Table 10, wherein it can be observed that the performance of UCIQE was better than that of UIQM when the fifteen TABLE 6. Image quality evaluations for yellowish underwater images in Fig. 6   TABLE 7. Image quality evaluations for greenish underwater images in Fig. 7.
underwater images were used for testing.The MOS value was obtained by using the subjective underwater image quality TABLE 8. Image quality evaluations for whitish underwater images in Fig. 8. evaluation procedure, which was proposed in our another work [123].
In summary, the accuracy of the state-of-the-art underwater image quality evaluation methods was not satisfactory due to the complexity of imaging environment of the underwater image (there was a lighting source) and degradation types (color deviation, lower contrast and noise, blurring, etc.).In particular, the authenticity of color restoration and the degree of detail restoration in dark areas were not in line with the quality evaluation criteria of subjective visual judgment.The average execution time of the UCIQE, UIQM and PCQI for underwater images shown in Table 11.The size of the test image was 778 × 1037 × 3, and tests were conducted on 3.2 GHz frequency Intel i5 double-core CPU and 8GB of RAM by using Matlab 2012b software.The data in Table 11 shows that UCIQE has the fastest execution speed, and it is applicable to the real-time underwater applications.The download links for some available codes are shown in Appendix B.

IV. DISCUSSION
In the future research on underwater image processing, researchers should consider the following aspects to carry out relevant work.

A. ALGORITHM ADAPTIVITY
The comparison and analysis presented in this paper prove that a satisfactory result can be obtained by adopting an appropriate enhancement method for various underwater tasks and environments.The ideal algorithm should be able to analyze the information of the input underwater image automatically, and make an adaptive adjustment for different scenes and lighting conditions to meet the requirements of complex situations.There is still a lack of research on the selection of an appropriate underwater enhancement method.In addition, the influence of uneven illumination from artificial lighting sources is less discuss.Besides, motion blurring is a degradation which exists in almost every underwater image, but it is rarely considered in enhancement or restoration methods.

B. BEYOND WORKING WITH SINGLE IMAGE
The research on the underwater video processing needs to be expanded; namely, most researches focus on a single underwater image and pay little attention to underwater video  processing, but the underwater video processing has a crucial role in practical applications.Presently, there are many problems which need urgent solutions: for instance, underwater video processing efficiency and inter-frame consistency need to be addressed.

C. UNIVERSALITY OF OBJECTIVE QUALITY EVALUTION OF UNDERWATER IMAGE
The contrast and partial color enhancements cannot be correctly evaluated by the existing underwater image quality evaluation methods.Establishment of a significant standardized objective evaluation method for underwater image enhancement is a challenge.Although the existing natural image databases play an important role in advancing the field of image quality prediction, image distortion in these databases is either single distortion simulated manually or distortion of an image taken by mobile devices.The images in the databases are scarcely underwater images.Furthermore, the performance of an image quality evaluation method based on the training using only one database is often poor when that method is applied to another database.However, it is very difficult to collect all kinds of distorted underwater images at all levels to produce meaningful evaluation results since underwater images are taken in an environment that is uncontrollable and unpredictable.The deep learning offers a potentially powerful framework for achieving sought-after gains in performance.However, the deep learning progress is limited by a lack of adequate amount of distorted picture data and ground-truth subjective quality scores.To the best of authors' knowledge, currently, there is no subjective quality benchmark database for underwater images.The measurement of color image enhancement or restoration results for different underwater assignments is difficult but important for automatic and real-time underwater processing.

V. CONCLUSION
In this paper, the existing methods for underwater image enhancement and restoration were introduced and the com-VOLUME 7, 2019 123653 mon problems in these methods were summarized.The effects of the typical underwater image enhancement and restoration methods on blueish, greenish, yellowish, offshore, and deep-sea images were compared, which provided a reference for the selection of most suitable method for underwater image enhancements under various cases.Besides, the limitations and accuracy of the widely-used underwater image quality evaluation metrics were analyzed.We also summarized the mostly used underwater image datasets and suggested possible research directions for future research.

APPENDIX A
See Table 12.

APPENDIX B
See Table 13.

FIGURE 2 .
FIGURE 2. The background light estimation results obtained by different restoration methods.From left to right: Input image, the output of the method of Carlevaris-Bianco et al.[38], the output of the method of Galdran et al.[40], the output of the method of Peng and Cosman[49], the output of the method of Li et al.[47], and the output of the method of Yang et al.[58].

FIGURE 5 .
FIGURE 5. Comparison of the results for blueish underwater images.(a) The original images.(b) The results of the method of He et al. [33].(c) The results of the method of Galdran et al. [40].(d) The results of the method of Peng and Cosman [49].(e) The results of the method of Li et al. [50].(f) The results of the method of Yang et al. [58].(g) The results of the ACE method [122].(h) The results of the method of Iqbal et al. [79].(i) The results of the method of Fu et al. [82].(j) The results of the method of Li et al. [87].

FIGURE 6 .
FIGURE 6.Comparison of the results for yellowish underwater images.(a) The original images.(b) The results of the method of He et al. [33].(c) The results of the method of Galdran et al. [40].(d) The results of the method of Peng and Cosman [49].(e) The results of the method of Li et al. [50].(f) The results of the method of Yang et al. [58].(g) The results of the ACE method [122].(h) The results of the method of Iqbal et al. [79].(i) The results of the method of Fu et al. [82].(j) The results of the method of Li et al. [87].

FIGURE 7 .
FIGURE 7. Comparison of the results for greenish underwater images.(a) The original images.(b) The results of the method of He et al. [33].(c) The results of the method of Galdran et al. [40].(d) The results of the method of Peng and Cosman [49].(e) The results of the method of Li et al. [50].(f) The results of the method of Yang et al. [58].(g) The results of the ACE method [122].(h) The results of the method of Iqbal et al. [79].(i) The results of the method of Fu et al. [82].(j) The results of the method of Li et al. [87].

FIGURE 8 .
FIGURE 8. Comparison of the results for whitish underwater images.(a) The original images.(b) The results of the method of He et al. [33].(c) The results of the method of Galdran et al. [40].(d) The results of the method of Peng and Cosman [49].(e) The results of the method of Li et al. [50].(f) The results of the method of Yang et al. [58].(g) The results of the ACE method [122].(h) The results of the method of Iqbal et al. [79].(i) The results of the method of Fu et al. [82].(j) The results of the method of Li et al. [87].

FIGURE 9 .
FIGURE 9. Comparison of the results for deep-sea underwater images.(a) The original images.(b) The results of the method of He et al. [33].(c) The results of the method of Galdran et al. [40].(d) The results of the method of Peng and Cosman [49].(e) The results of the method of Li et al. [50].(f) The results of the method of Yang et al. [58].(g) The results of the ACE method [122].(h) The results of the method of Iqbal et al. [79].(i) The results of the method of Fu et al. [82].(j) The results of the method of Li et al. [87].

TABLE 1 .
The DCP based underwater restoration methods sorted by year.

TABLE 2 .
Underwater image enhancement methods sorted by year.

TABLE 3 .
Deep learning based underwater enhancement methods.

TABLE 4 .
List of underwater image datasets

TABLE 9 .
Image quality evaluations for deep sea images in Fig.9.

TABLE 10 .
Correlation between the underwater image quality evaluation metrics and mos.

TABLE 11 .
Average execution time of the UCIQE, UIQM and PCQI.

TABLE 12 .
Underwater image defogging based on DCP.

TABLE 13 .
Download links for some codes.