Medical Image Coloring Based on Gabor Filtering for Internet of Medical Things

Color medical images better reflect a patient’s lesion information and facilitate communication between doctors and patients. The combination of medical image processing and the Internet has been widely used for clinical medicine on Internet of medical things. The classical Welsh method uses matching pixels to achieve color migration of grayscale images, but it exists problems such as unclear boundary and single coloring effect. Therefore, the key information of medical images after coloring can’t be reflected efficiently. To address this issue, we propose an image coloring method based on Gabor filtering combined with Welsh coloring and apply it to medical grayscale images. By using Gabor filtering, which is similar to the visual stimulus response of simple cells in the human visual system, filtering in 4 directions and 6 scales is used to stratify the grayscale images and extract local spatial and frequency domain information. In addition, the Welsh coloring method is used to render the image with obvious textural features in the layered image. Our experiments show that the color transboundary problem can be solved effectively after the layered processing. Compared to images without stratification, the coloring results of the processed images are closer to the real image.


I. INTRODUCTION
With the development of modern communication technology, telemedicine using mobile devices or the Internet has developed rapidly. We can even conduct telemedicine consultations through mobile devices at home. Most medical images used in Internet of Medical Things are grayscale images in which lesion information is quickly recognized by experienced doctors [1], [2]. However, it is difficult for patients and residents to interpret medical images such as x-rays, CT scans and MRIs. By applying image coloring technology to grayscale medical images, colored medical images can better reflect the areas of clinical focus of the patients, identify key medical information more effectively The associate editor coordinating the review of this manuscript and approving it for publication was Wei Wei . and quickly and facilitate communication between doctors and patients. With the development of Internet technology, the influence of the Internet on medical treatment is increasing, and the use of telemedicine is becoming a trend [3], [4]. Mi jing et al. were responsible for introducing key technologies from the Internet of things and also the development of medical education under the Internet of things. Currently, there is an urgent need to build a smart medical systems based on the Internet of things [5], [6]. In order to enable telemedicine, Xiao Yang et al. proposed constructing an Internet of things and service-oriented architecture system on a technical base, with four added layers in the system design, namely, the processing layer, transport layer, network layer and application layer [7], [8]. Presently, graphics and image processing are widely used in medicine. Huabin et al. proposed an improved color fusion method for medical images.
A KNN-based image pre-background discrimination algorithm is used to enhance the boundary information of the lesion area, and users can obtain better coloring results with a simple input [9]. Li qiang et al. conducted segmentation experiments on brain tumor images that focused on a method of brain tumor segmentation based on deep learning [10].
Presently, due to its rapid development, image coloring is playing an increasingly important role in medical image processing. This paper divides existing image rendering methods into three categories. The first category is image coloring methods based on deep learning. By training models using deep neural networks and color image data, we can render grayscale images based on a model. He M et al. proposed that convolutional neural networks can directly map grayscale images into color images [11]. Unlike the traditional example-based approach, an end-to-end coloring network can learn how to select, propagate, and predict colors from large-scale data. The model is robust and generalized even when the reference images are not related to the grayscale image inputs. However, the dataset must be normalized before training, and it is difficult to train different sizes of images mixed together. Yuki E et al. proposed an edit propagation method [12]. Without manual feature selections, effective editable propagation features can effectively learn DNN from sparse user inputs from a single image and avoid getting trapped in bad local solutions due to the vanishing gradient problem.
The second is an image coloring method based on user interaction. This method has good interactivity and control. Levia A believes that the brightness of an image is related to its color [13]. If the brightness is similar, the color is similar. Based on this idea, a framework was proposed in which users sketch a few strokes of color on the image and then automatically complete the coloring of the remaining pixels. The color choice of the sketch can cause very different coloring results, so the coloring result is dependent on the user's skills and experience. However, when the user marks the boundary of an image, it will produce an unrealistic effect. Li Hong-an et al. constructed an image color diffusion model based on the linear relationship between the brightness signal and the color signal of the local block image [14]. In the modeling process, the linear coefficients of brightness and color are used as the block image features instead of the pixel information. This ensures that the brightness of the image does not participate in the modeling operation, and the complete content of the image is retained [15].
Image coloring methods based on color transfer is the third image coloring category. The color transfer method proposed by Reinhard E et al. has the advantages of a simple implementation and high operational efficiency [16]. This method is an overall color transfer, so it has a good effect on images with a single global color tone. However, for color content rich images, the effect is not so obvious. Welsh T transfers the features of the source image to the target image by converting the target image and sample image into the Lab color space, matching the texture information and brightness information, to achieve the coloring of the grayscale images [17].
Most of the existing image coloring methods are difficult, entail time-consuming processes, and are not suitable for processing large quantities of data. The Welsh method can render images quickly and easily, but the coloring results have color overflow problems. Therefore, to remedy these issues, we propose an image coloring method based on Gabor filtering. When Gabor filtering is applied to an image to be rendered it makes the textural features of it more obvious and reduces the color overflow. Experimental results show that image coloring using Gabor filtering can effectively suppress the color overflow phenomenon, and the resulting image's structural similarity and peak signal-to-noise ratio are closer to the real image. In this paper, real medical CT images are selected for coloring. Experiments show that the proposed method can retain the structural information of the original image more effectively and produces high quality images after coloring.
The structure of this paper is as follows: Section I introduces applications of the Internet of Medical Things and the existing image coloring methods. Section II introduces the characteristics of the Gabor filter and the application of traditional Welsh image coloring methods for medical image processing. In Section III, we propose an image coloring method that combines the Gabor filtering method and the Welsh method. Experiments and an analysis are presented in Section IV. Section V concludes this paper.

II. RELATED WORK A. THE GABOR FILTER
Two-dimensional Gabor filtering is very similar to the visual stimulus response of simple cells in the human visual system. Sheng Wen defined the two-dimensional Gabor function as a complex sine function modulated by a Gaussian function whose scale parameter is σ , aspect ratio is λ, and where the X-axis is at an angle with the main axis [18]. Its expression is written as follows: where, (x , y ) = (x cos φ + y sin φ, −x sin φ + y cos φ) is the rotation coordinate and two-dimensional Gaussian function is defined as: In the Fourier transform formula (1), the frequency-domain expression of the two-dimensional Gabor function is as follows: In the formula (u , v ) = (u cos φ+v sin φ, −u sin φ+v cos φ), (U , V ) is a similar rotation made by the central frequency (U , V ).

VOLUME 8, 2020
Multi-angle and multi-directional Gabor filtering plays an important role in face recognition and texture extraction [19], [20]. Gabor filtering at different scales in the same direction is shown in Figure 1.  Gabor filtering in different directions at the same scale is shown in Figure 2. Experiments show that when the total number of directions is four and the total number of scales is six, the correlation between Gabor filters is the smallest, which minimizes the redundant information generated by filtering images. Using different directions and scales of a Gabor filter to analyze the textural information of an image allows us to distinguish the same textures between different scales and different directions. There are a large number of images in the Internet of Medical Things, and the effect of 4-direction and 6-scale Gabor filtering on an abdominal CT image filtering is shown in Figure 3.

B. THE WELSH METHOD
The Welsh method processes the grayscale image as a onedimensional distribution, thus only the brightness channel can be matched between the two images. Since a single luminance value can represent different parts of the image, the statistical information in the pixel neighborhood is used to guide the matching process. When the pixels are matched successfully, the original brightness value is retained. After the color is transferred between the image to be shaded and the reference image, each pixel of the grayscale image to be shaded is matched with the pixel in the reference image by using the L 2 distance metric, and the final color is assigned to each pixel in the grayscale image to be shaded. Each pixel match is determined by matching each pixel only to other pixels in the same image [21]. The error distance E of L 2 is defined as formula 4: where I is the grayscale image, S is the brightness channel of the color sample, and P is the domain pixel of the grayscale image.
The specific coloring steps are as follows: Step 1. Convert the reference image and the image to be shaded from the RGB color space to the lαβ color space.
Step 2. Calculate the brightness mean (L s , L r ) and standard deviation (σ s , σ r ) of the grayscale image (g) to be colored and the reference image (r).
Step 3. The brightness of the image to be colored is linearly transformed so that the mean value and variance of the brightness of the image to be colored and the reference image are consistent, so as to prevent matching errors caused by too large a brightness difference. The result after transformation L s is as follows: Step 4. Calculate the neighborhood variance (σ w ) of each pixel of the image to be colored and the reference image, and the weighted sum of the luminance value (L) of this pixel to obtain the image L 2 distance, that is: Step 5. Calculate the L 2 distance of each pixel point of the image to be colored and the reference image, find the sample point with the smallest distance from the L 2 distance, and finally assign α and β channel values of the sample point of the reference image to the corresponding pixel point of the image to be colored to achieve the color transfer.
Step 6. The image is converted from lαβ space to RGB space, and the color migration ends. The coloring result of the Welsh method is shown in Figure 4. As is shown in Figure 4 (c) and Figure 4 (d), when coloring a patient's abdominal CT image, in comparison with directly coloring the image with the Welsh method, the texture of the image after Gabor filtering is clearer, the phenomenon of color overflow can be effectively suppressed and the pathological information of the patient is better retained.

III. LAYERED IMAGE COLORING METHOD BASED ON GABOR FILTER
To solve the problems of color overstepping and boundary blurring after rendering with the Welsh image rendering method, we propose an image rendering method that combines the Gabor filtering method and the Welsh method. The frame diagram of the method proposed in this paper is shown in Figure 5, and can be divided into two steps. The first step is to use a Gabor filter to generate an appropriate texture map, and the second step is to use the Welsh method to color the filtered texture map.
Firstly, we utilize a Gabor filter to process the image, which improves the effect of uneven illumination and image noise on the image rendering. Based on the multi-scale and multi-directional characteristics of the Gabor filter, 24 texture images with four directions and six scales were obtained after the grayscale image g was preprocessed. Twenty-four texture images were respectively colored as input images, and the texture image w with the best direction and scale was obtained as the final image to be colored.
Next, since the Welsh method is based on matching colors by brightness among the images, we need to perform a color space conversion, which involves the conversion from an RGB color space to a color space. One color space cannot be directly converted to another color space. The XYZ color space is required to convert the RGB color space to the XYZ color space, and then from the XYZ color space to the color space. The conversion formula between the RGB color space and the XYZ color space is presented in Formula (7). where, R, G, B respectively represent the values of each color component in the RGB color space, and X, Y and Z respectively represent the component values in the XYZ space. The formula for converting the XYZ color space into the lαβ color space is as follows: where, X n , Y n , Z n are the values of the XYZ trichromatic stimulus with reference to the white point CIE, X n = 96.422, Y n = 100.000, Z n = 82.522. The specific steps of the layered image coloring method based on the Gabor filter are as follows: Step 1. Four angles and six scales are used to perform Gabor filtering on a grayscale image (g), and 24 filtered images are generated. VOLUME 8, 2020 Step 2. Determine the image with the clearest boundary information and texture information and designate it as the final image (w) to be colored.
Step 3. Convert image (w) from the RGB color space to the lαβ color space. Calculate the brightness mean value and standard deviation of image (w) and image (r).
Step 4. Carry out the linear transformation on the image (w), so that the mean value and variance of the image (w) and image (r) are consistent in order, to prevent color matching errors.
Step 5. Calculate the neighborhood variance of each pixel point of image (w) and image (r), and use the weighted sum with the brightness of this pixel point to obtain the image distance. Finally, find the sample point with the smallest difference from the distance, and then, assign the values of the r sample point and the channel of the reference image to the corresponding pixel points of the image to be colored to achieve the color transfer.

IV. EXPERIMENT AND ANALYSIS
In order to test and evaluate the coloring result based on the Gabor filter with the Welsh coloring compared with the direct coloring result, this paper uses an experimental group and a control group for coloring. The experimental group colors the images after Gabor filtering, while the control group colors the images directly. The experiment was run on a computer with a 64-bit Windows 10 operating system, Intel(R) Core(TM) i5-6300HQ CPU @ 2.30GHz processor, 8G memory, NVIDIA GeForce 960MX graphics card, and the computer was equipped with Python 3.6 and pycharm. Internet of Medical Things involves professional communication, sharing and collaboration, and we use the medical images which derived from real cases to carry on our experiment. This process also supported DICOM medical imaging and assisted Internet of Medical Things [22].

A. GABOR FILTERING
For the experimental group, we used Gabor filtering in 4 directions and of 6 scales to filter the images. The four directions are 0 • , 45 • , 90 • , 135 • , and the six scales are 1, 2, 3, 4, 5, 6, respectively. It was found that the Gabor filter had the clearest texture and boundary information when the angle was 0 • and the scale was 1, and it was able to obtain best coloring effect. Experimental results show that the detailed description and outline of the CT image in Figure 6 was clearer after filtering.

B. COMPARISON OF COLORING IMAGES
The reference coloring image used in the experiment is shown in Figure 7. The colored CT image is shown in Figure 8 because the texture of the image was clearer after Gabor filtering. The Welsh method can better match a real pixel when matching pixels. Because the boundary texture of images without Gabor filtering was not clear, it caused the color overflow of the image. The results show that the coloring of the image after Gabor filtering has better performance.
PSNR [23] is an objective evaluation metric to estimate image quality. It is generally used for engineering projects between maximum signal and background noise. The higher the PSNR, the higher the transmission efficiency. The calculation formula is as follows: where, H , W represent the width and height of the image, (i, j) stands for pixel coordinates, n is the bit of pixels.
In order to measure the image quality of the improved coloring model that has been enhanced, we use PSNR to evaluate the coloring image with or without Gabor filtering. For the image in Figure 8, the average result of our method is shown in Table 1.
The Gabor filter is a band-pass filter, it only allows the texture corresponding to its frequency to pass, so the energy of other textures is suppressed. The Welsh method has smaller errors when comparing the corresponding pixel  of the real image. Therefore, the texture of images is clearer after Gabor filtering and the color is closer to the real image. The average PSNR score is 0.21 higher. It can be seen from the table that the Welsh coloring method based on the Gabor filter can obtain better image quality, and the overall coloring quality of the image improves.
SSIM is a kind of image structure similarity evaluation metric at full reference, and comparison steps are shown in Figure 9. It measures image similarity from brightness, contrast and structure. The value range of SSIM is [0, 1]. The larger the SSIM value, the smaller the image distortion.
The formula for an image brightness comparison is as follows: The formula for an image contrast comparison is as follows: The formula for an image structure comparison is as follows: VOLUME 8, 2020 The formula for SSIM is as follows: where, µ x , µ y represent the average value of the real image and the coloring image, and δ 2 x , δ 2 y is the variance of the real image and the coloring image respectively. C 1 , C 2 , C 3 are constants used to avoid systematic errors when the denominator is 0. δ xy is the covariance of the real image and the coloring image. Using the same method to analyze the SSIM as PSNR, the evaluation results of the image model in Figure 8 are shown in Table 2. The Gabor filter can well describe the local structure information corresponding to spatial frequency, spatial location, and direction selectivity. It retains the details of the image, and is insensitive to light changes. Comparing it to the real image, the ratio of brightness, structure, and contrast are closer to 1. The coloring image is most similar to the real image. It can be seen from the table, that the improved model in this paper enhances SSIM by 5 percent, which better maintains the structural information of the original image.
Our method not only has a good effect on medical image processing, but also is suitable for general image processing. For the general image experimental group, 4 × 6 Gabor filter was used for filtering processing. The four directions are 0 • , 45 • , 90 • , 135 • , and the six scales are 1, 2, 3, 4, 5, 6. The Gabor filter has the clearest texture, boundary information, and best color effect when the angle is 0 • and the scale is 1. Figure 10 is a reference color image, and Figure 11(a) is an image to be colored. Figure 11(b) is selected as the input  image to be colored after Gabor filtering, and the coloring effect is shown in Figure 11(d). After filtering, the colored image can effectively retain the image boundary information, and suppress the phenomenon of color overflow.
For the petal coloring in Figure 12, because the texture of the image is clearer after Gabor filtering, our method can match the real pixels better. The direct coloring method resulting in color overflow because of unclear boundary texture. It can be seen that the color effect of the image after Gabor filtering is better. To reflect the improvement of image quality of the improved image coloring model more realistically, PSNR is used to evaluate the images generated by the Gabor filtering and direct coloring.
Because the texture of the image filtered is clearer, the color is closer to the real image, and the average PSNR score has improved by 2.2. As shown in Table 3, our Welsh coloring method based on Gabor filter has better quality of the colored image.   As the same as PSNR analysis, SSIM metric for the above images are shown in Table 4. Our method has improved SSIM index by 8 percentage points, which can better maintain the structural information of the original image.

V. CONCLUSION
In order to better render images for medical image processing, and to make the coloration of grayscale medical images closer to that of the real image to the greatest extent, this paper presents a coloring method based on Gabor filtering and Welsh coloring. The advantages of the Gabor filter and the Welsh coloring method are put forward. Our proposed method uses a Gabor filter to perform hierarchical filtering on the original image, and selects the image with the optimal textural features as the image to be colored. Then, the Welsh method, using its simplicity and speed, renders the grayscale image. Experiments on a large number of actual medical images show that this method can extract textural information effectively, which improves the color, overflow and color monotony caused by direct coloring.
In addition, a comparison experiment and analysis of multiple images show that the color image after Gabor filtering is greatly improved for peak signal-to-noise ratio and structural integrity. The method proposed in this paper produces images closer to the real image. MIN ZHANG received the bachelor's degree in information and computing science from the College of Computer Science and Technology, Xi'an University of Science and Technology, in 2019, where she is currently pursuing the master's degree.
Her research interests include the use of image processing and computer vision, and she did a small project on ''Research and implementation of image classification based on the GoogleNet model.'' QIAOXUE ZHENG received the bachelor's degree in information and computing science from the College of Computer Science and Technology, Xi'an University of Science and Technology, in 2020, where he is currently pursuing the master's degree.
His research interests include visual media computing and artificial intelligence. VOLUME 8, 2020