Automatic Segmentation of Crop/Background Based on Luminance Partition Correction and Adaptive Threshold

Crop segmentation is a fundamental step of extracting the guidance line for an automated agricultural machine with a visual navigation system. However, the segmentation quality of green crop is seriously affected by the outdoor lighting conditions. To improve the accuracy of crop segmentation under complex lighting conditions, a color-index-based crop segmentation method with luminance partition correction and adaptive thresholding is proposed in this article. The method extracts the luminance component from the given RGB image and employs two adaptive thresholds to divide the luminance image into the dark, normal and bright areas. Then, a partition Gamma function is adaptively selected to improve the brightness levels of the dark and bright regions in which the Gamma correction parameter is adaptively determined based on the local distribution characteristics of illumination, and the corrected image is converted to the RGB counterpart through color saturation restoration. Finally, the ExG (excess green index) color index with Otsu thresholding is used to perform pre-segmentation in order to calculate the segmentation threshold for the final segmentation. Experimental results show that the proposed approach can effectively increase the brightness levels of the dark region and decrease the brightness levels in the bright region. In addition, compared with the traditional Otsu method employed in before and after luminance correction, the proposed method can achieve better segmentation results.


I. INTRODUCTION
Automated navigation technology is important for advancing modern agriculture and has been widely used in transplanting, weeding, fertilizing and harvesting to reduce labor intensity as well as improving operation efficiency and safety [1]. To achieve autonomous navigation, the significant step is to provide accurate navigation information with the positioning sensors which mainly include GPS and cameras [2]. Although GPS provides high-precision positioning information of agricultural machines, its performance largely depends on the availability and quality of GPS signals which are easily blocked by buildings and trees, thus reducing positioning accuracy. Compared with GPS, visual navigation systems can segment crops and extract the navigation lines in relation to The associate editor coordinating the review of this manuscript and approving it for publication was Naveed Akhtar . crop rows, allowing the machinery to determine its position relative to the navigation line and follow crop rows without damaging crops. Thus, visual navigation systems have been widely used in agricultural production.
In visual navigation, crop segmentation usually is the first and essential step towards the guidance line extraction, which aims to classify different pixels in images into plant and background classes. The color-index-based method is one of the commonest methods applied in this task, in which the RGB color space is firstly converted into alternative color spaces based on color indexes including ExG (excess green index) [3], ExGR (excess green minus excess red index) [4] and COM (combined index) [5], etc. These color indexes result in a gray image with accentuating certain wavelength of an image, i.e., with pixels brighter for plant pixels and darker for non-plant pixels, and then a pre-defined threshold is used to segment plants [6]. However, the uncontrolled lighting VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ conditions in the field directly affect the color of plants which can cause the index-based methods failure to accurately separate plants from the background. In order to improve the performance of the index-based methods, some studies perform plant segmentation in other color spaces. Raja et al. [7] proposed a method for weed-crop recognition from the image taken under ultraviolet and white light illumination by using the HSV color space, and the YCrCb color space was used in [8] to segment rice seedlings with high accuracy. Studies have shown that incorrect segmentation caused by illumination variation can be reduced in these color spaces, but they are not robust to all possible illumination conditions [9], especially in the presence of uneven illumination or highlight. Moreover, they will fail to achieve the desired segmentation result without an optimal threshold. In addition to optimal color space selection, image enhancement as a pre-processing step is used for enhancing and adjusting the contrast of the acquired image to solve the varying illumination issues. Many frequently used image enhancement techniques are available, including methods based on histogram equalization, retinex theory and Gamma correction. Chang et al. [10] applied CLAHE (contrast-limited adaptive histogram equalization) to enhance the brightness of all the images. Sun et al. [11] used MSRCR (multi-scale retinex with color restore) to reduce the influence of natural lighting on the color characteristics of cucumber canopy images before segmentation. To improve image visibility, the captured coffee leaf is processed for contrast enhancement using Gamma correction [12]. Although the above-mentioned studies have showed that image enhancement algorithms produced better segmentation performance compared with the methods without any preprocessing, the enhancement results are related to the parameter settings used in the automated enhancement methods. With fixed parameters, different images or areas of non-uniform illumination images tend to exhibit similar changes in intensity. Further, color distortion can occur when the enhanced image contains extreme contrast [13]. Therefore, higher luminance correction quality is required to process crop images based on the index-based applications.
Different from the color-index-based methods, learningbased techniques like decision trees [14] or deep learning [15] have been proposed and used in several applications. Although these methods can obtain good crop segmentation results in a complex environment, their performance relies on whether the training samples cover the characteristics of different illuminations [16]. Since the change of illumination happens all the time without rules and regulations, the training samples are limited and the classification results cannot be guaranteed. Furthermore, machine learning techniques are bound by the training data and may not work under different crop types when this was not considered in the training data. One way to overcome this issue is to use transfer learning available for deep network training where the network is trained on one dataset and tested on another [17]. In [18], it is mentioned that the method based on transfer learning will also lead to poor results if the light is too weak or bright. In addition, the manual labeling numerous crops in images is a tedious and laborious work, and easily causes inaccurate labeling of crop boundaries, thus deteriorating the performance of the trained network.
Compared with learning-based methods, index-based approaches have certain advantages. Firstly, color is an intuitionistic and easy characteristic to distinguish plants from the background. Secondly, the color-index-based method is independent of numerous annotation labels. Therefore, the aim of this work is to develop a crop segmentation approach combining the advantages of index-based methods with image enhancement to be more robust to outdoor illumination conditions. Considering that the sensitivity of HSV to luminance, we convert the RGB image into the HSV counterpart in which the V component is divided into the dark, bright and normal brightness areas based on two adaptive thresholds. Subsequently, partition Gamma correction with adaptive correction parameter is performed to dynamically adjust the brightness levels of the dark and bright areas. Then, the local fusion and color saturation restoration operators are used to reduce sharp changes of the brightness and color shift of the corrected image. Additionally, the corrected image is transformed to the RGB color image and an index-based threshold method is introduced to finalize crop segmentation. Experiments on several test crop images were conducted to demonstrate the effectiveness of the proposed method.
The remainder of the paper is organized as follows. Section II briefly analyses the index-based segmentation and Gamma correction. Section III presents the proposed method in detail. Section IV shows the experimental results, and we conclude this article in Section V.

II. ANALYSIS OF INDEX-BASED SEGMENTATION AND GAMMA CORRECTION
The outdoor illumination has a decisive influence on the segmentation quality of green crops due to its properties with uncertainty and instability. Fig. 1 shows the segmentation results of crop images under different illuminations. The first column shows the original images taken under different natural illuminations (sunny or cloudy conditions). In sunny day, as shown in Fig.1 (a), specular reflection produces highlight spots in the crop or background. The highlight spots occurring in the crop are tagged by red cycles in the original RGB image. In cloudy day, as shown in Fig.1 (d), the green component of the plant is reduced owing to the lack of enough illumination and the color of the background is similar to that of the crop. The color index ExG is applied in image graying process to accentuate the green chrominance of the crop region. The corresponding transformed images are shown in the second column, where we observe that the green chrominance of the plant is affected by the highlight spots or the dark illumination and the affected plant pixels are not highlighted. The gray-level images shown in Fig.1 (b) and (e) are processed with the Otsu method [19], and the segmentation results in the RGB format are shown in Fig.1 (c) and (f), where the background is identified and marked as black pixels. From Fig.1(c) and (f), it can be seen that plant pixels are misclassified into the background labeled with red cycles when highlight occurs or the contrast between plants and the background is not obvious. Hence, it is necessary to correct image luminance to improve the performance of crop segmentation under outdoor illumination.
For adjusting image luminance, Gamma correction is one of the most commonly applied methods. Its general formulation of Gamma correction can be written as follows: where l in and l out are the intensities of the input and output images, respectively. l max is the maximum intensity of the input. γ is the parameter which directly determines the correction effect. With a small γ value (γ < 1), Gamma correction benefits improving the brightness of the dark regions in the images, while a large γ value (γ > 1) can help recover the details of the bright regions. Fig. 2 shows the Gamma correction results of the uneven illumination image with different correction parameters. As seen in Fig.2 (a), two points in the rice seedling image are selected to explore Gamma correction preformation, where the pixel ''A'' is located in the dark region and the pixel ''B'' is located in a highlight region. Gamma correction with γ (x, y) = 0.5 is selected to improve the visibility and a higher value γ (x, y) = 1.5 is selected to reduce the brightness levels. The correction results and gray intensity profiles are demonstrated from Fig.2 (b)-(f). It can be seen that the brightness level at pixel ''A'' can be improved with different γ (x, y) values, i.e., the pixel value is brightened from 42 to 107 when γ (x, y) = 0.5. Unlike pixel ''A'', the brightness level at pixel ''B'' remains the same before and after correction (see Fig.2 (d)-(f)). Therefore, selecting an adaptive parameter γ (x, y) is important to correct luminance in both the dark and bright regions of the image. Huang et al. [20] presented an adaptive Gamma correction method (AGC) where parameter γ (x, y) is adjusted based on the probability density function (PDF) of the intensity histogram. However, global Gamma correction usually achieves luminance correction by a global mapping without considering local information, which cannot adaptively correct the luminance of a special image or a specific region of the image, especially in the highlight region with pixel value 255. To this end, the key point is to require a correction operator based on luminance partition and select appropriate correction parameter adapting to different scenes or regions of the given image.

III. THE PROPOSED METHOD
In this work, we propose a strategy for setting adaptive correction parameter and performing luminance correction to adaptively adjust the luminance value in bright and dark regions. Then, the corrected image is used to support the color-index-based method with adaptive thresholding. The flowchart of the proposed method is shown in Fig.3.

A. IMAGE LUMINANCE CORRECTION
As shown in Fig.3, the luminance component is first extracted by transforming the image from the RGB space to the HSV space which is sensitive to luminance changes VOLUME 8, 2020  in images. Considering that an image usually contains low and high-brightness regions, especially in the outdoor fields, the luminance image is partitioned into three areas with different brightness levels. Then the illumination information of luminance image is estimated through a fast guided filter, and two appropriate Gamma functions are built to adjust the luminance of dark and bright regions of the input image, respectively, where the Gamma correction parameters are adaptively selected based on the characteristics of the illumination distribution. Finally, the Gamma correction luminance image is fused by a filtering operator, and transformed to the RGB space with an adaptive color saturation restoration.

1) LUMINANCE PARTITION
To separate the luminance information from the given image, the image is transformed from the RGB space to the HSV space, in which the V component of HSV is luminance information. In this work, we divide the whole luminance image into three brightness levels, namely dark, bright and normal areas. Specifically, the minimum and maximum intensity 202614 VOLUME 8, 2020 values in the V component are firstly obtained, which are denoted as V min and V max , and the mean intensity value V avg is calculated to divide the image into two parts. Then, we calculate the mean and standard deviation values of intensities in the first half (i.e.. pixels with V (x, y) ≤ V avg ) and the second half(i.e.. pixels with V (x, y) > V avg ), and set two partition thresholds V 1 and V 2 according to the Chebyshev probability inequality shown as follows: where µ d , µ b , σ d and σ b represent the mean and standard deviation values of intensities within the two parts, respectively. V 1 is utilized to segment the first half into two parts, namely dark and normal areas. The second half is also segmented into two parts by V 2 , namely normal and bright areas. Consequently, the image V is segmented into the following three brightness levels: which represent the dark, normal and bright luminance areas, respectively, and the pixel sets of three regions are given by  4 shows the luminance partition results of three images under different illuminations. The black, gray and white regions in the second row represent the dark, normal and bright brightness areas generated by luminance partitioning, respectively. The histograms of the partition results are shown in the third row. As shown in Fig. 4, it can be observed that the partition results are highly consistent with the human visual observations, which can provide support for partition luminance correction.

2) ADAPTIVE GAMMA CORRECTION
For various images, their intensity ranges vary largely. To avoid inconsistency, a logarithm transform or maximum normalization method is used to normalize the luminance image with the general Gamma function [21]. According to (1), the intensity value of the pixels with V (x, y) = 255 after being normalized always is 1 resulting in that the intensity value of these pixels cannot be changed before and after correction. However, images taken in the outdoor fields usually contain high brightness areas. In this work, the weighted inverse processing method is applied in the highlighted area to avoid the problem that the brightness levels of the highlighted pixels cannot be corrected, and two adaptive Gamma functions are selected to separately adjust the brightness levels of the dark and bright regions of the same image, which are defined as follows: where V d (x, y) and V b (x, y) are respectively values of the pixel intensity after Gamma correction in dark and bright areas. The inverse operator may severely reduce the intensity value in the bright areas. Therefore, as shown in (6), a weighted process is given before and after Gamma correction, and w is set to 0.4. For parameters γ d (x, y) and γ b (x, y), they are obtained based on the local distribution characteristics of illumination. The luminance image V is filtered using a fast guided filter [22] to estimate its illumination L(x, y), shown as: where W x,y is a square window centered at the pixel (x, y) with radius r = 16, and k is the index of W x,y ; a xy and b xy are coefficients of constant and linearity in W x,y ;V k is the mean value of intensity in W x,y ; µ xy and σ xy are mean and variance of the guided image in W x,y ; ε is a regularization parameter set to 0.001; L k (x, y) and G k (x, y) are respectively values of pixel k in W x,y of the output and guided images. In this work, the luminance component serves as the guided image. The reason is that the fast guided filter characterizes the low-frequency information of the input image through smoothing and has good performance in preserving high frequency edge details when the input image is taken as the guided image [22]. Then we divide the illumination image into three areas based on the above luminance partition, which are denoted as L d , L n and L b , and obtain two correction parameters by the local distribution characteristics of illumination as: where L avg is the mean value of the illumination component

3) LOCAL FUSION AND COLOR SATURATION RESTORATION
The brightness levels in the dark and bright areas of the image can be adaptively adjusted with two Gamma correction functions, but direct stitching of different correction areas may cause sharp changes of brightness in the boundary and even reduce the local contrast of some areas. Here, a local fusion operator based on the difference of Gaussian filter is used to improve visual effects near edges and enhance the details of the corrected image as follows: where V (x, y) is Gamma correction image. f (x, y, σ ) is a difference Gaussian function in which σ and m are set to 0.4 and 5 respectively in our work.
In above-mentioned Gamma correction based on luminance partition, intensity values in the dark and bright areas can be corrected, however this process tends to introduce changes in color appearance when recombining luminance values into a color image [23]. Hence, color correction is required to preserve color appearance after luminance correction. A common and efficient method [24] for addressing the color appearance issue is to control the color saturation by preserving three color ratios of three color channels (red, green and blue) before and after luminance correction, which is expressed as: where I represents one of the color channels; V is pixel luminance; in/out subscripts are pixels before and after correction; α is the color correction factor controlling color saturation.
According to the existing studies [25], under-saturation often happens in the dark areas of the image while the bright regions may be over-saturated, and the parameter α should be automatically adjusted based on the resulting luminance image. Based on these facts, we employ the color saturation restoration as follows: where c ∈ {R, G, B} denotes one of the color channels; I c (x, y) and I c (x, y) are pixels before and after color correction; V (x, y) and V out (x, y) are the pixel luminance before and after Gamma correction; α(x, y) is color correction factor and decreases with the increasing of luminance V out (x, y).

B. CROP SEGMENTATION
In this work, color-index-based method with adaptive thresholding is applied in crop segmentation, where the ExG index is used to transform a color image to a gray-level image with the enhanced contrast between plant pixels and the rest of the image. The ExG index is defined in Eq. (17).
where R, G, B denote the pixel values of the image in the RGB color space. After the ExG application, the transformation image generally consists of two main classes i.e. plant and non-plant pixels. For thresholding segmentation between two classes, the Otsu method is a widely used to calculate threshold value due to its simplicity and adaptability [26], which is given as Eq.(18) searching for the threshold at which the within-class variance is as small as possible, but between the classes at the same time as large as possible.
where p 1 (t) and p 2 (t) are cumulative probabilities, σ 2 1 (t) and σ 2 2 (t) are the variances of the two classes segmented by t, respectively. σ 2 w (t) denotes the within-class variance. The Otsu method can achieve good segmentation performance for the images with a bimodal histogram distribution. As mentioned in [27], the threshold T otsu is equal to the average of the mean values of the two classes which have a similar variance. Let µ 1 and µ 2 denote the mean levels of the plant and background classes. The Otsu's threshold is shown in Eq. (19). However, the threshold biases toward the class with the larger variance or a large size for images with a unimodal or near-unimodal histogram, which is the case for crop images. As shown in Fig.5, the green plant area is much smaller than the background area in the paddy fields. In Fig.5 (b), ground truth is labeled manually and employed as mask to get ideal segmentation of the ExG image. As seen in Fig.5 (c), the histogram of the segmented ExG image only has a near-unimodal distribution where the blue area represents the background pixels and the red area is the plant pixels. The pixel value at the boundary of the two regions serves as the ideal threshold locating at the green dash line as shown in Fig.5 (c), denoted by T ideal . We observe that the Otsu method tends to shift the threshold toward the background class and T ideal > T Otsu . Thus the Otsu's threshold T Otsu causes many background pixels to be misclassified into the plant pixels as shown in Fig.5 (d).
To solve the bias problem in the case of unbalanced distributions, we use the Otsu's method for pre-segmentation to approximately separate plant and background pixels, and then calculate the mean values of plant and background pixels in an ExG image. Based on the above analysis, the threshold T otsu should be located between these two means, while it wouldn't be exactly in the middle under the case of the  unbalanced distribution. The final threshold denoted by T seg will increase from T otsu when the background class is with a large variance, whereas T seg will decrease. According to Eq. (19), the final threshold T seg shown in Eq. (20) is determined in combination with the standard deviations of plant and background pixels. Fig. 6 shows the histogram of the ExG image from Fig.5 (a) and different threshold values. We can find that T seg obtained from Eq. (20) (referred to as the improved Otsu method) is closer to the ideal threshold T ideal compared with T otsu . Then, the final segmentation in the ExG image is performed with T seg . The proposed segmentation algorithm is described in Algorithm 1.
where σ 1 (T otsu ) and σ 2 (T otsu ) are the standard deviations of the plant and background pixels obtained from pre-segmentation.

IV. EXPERMINTAL RESULTS
In this section, we evaluate the performance of the proposed method with the test images collected from the VOLUME 8, 2020  (2) and (3):

A. COMPARISON OF DIFFERENT APPROACHES IN LUMINANCE CORRECTION
To validate the effectiveness of luminance correction proposed in our paper, four different correction methods including contrast-limited adaptive histogram equalization [10] (referred as CLAHE), the multi-scale retinex method [11] (named MSRCR), the adaptive correction algorithm [20] (called AGC) and the proposed method were used to correct the crop images. The visual comparison results are illuminated in Fig.7. From Fig.7 (b), it can be seen that CLAHE achieves good contrast enhancement, but it brings color distortion in the corrected images. From the results shown as in Fig.7(c), although MSRCR can increase the intensity of the image with poor illumination and decrease the intensity of the bright image, it introduces the noise information in the image thus losing the local details of the green plants. Compared with the MSRCR method, AGC can enhance the details of the corrected images and provide good contrast in the images with poor lighting, but the correction effect decreases obviously in the highlight area caused by water surface reflection and crop leaf surfaces with over-bright pixels (see Fig.7 (d)). For luminance correction with the proposed method in this work (see Fig.7 (e)), the luminance in the dark and bright areas can be well corrected, and the details of the input image can be preserved, especially in the highlight region with pixel value 255. The Gamma correction with adaptive parameters used in this work takes into account the luminance partition. Therefore, the proposed method outperforms the other three methods in luminance correction which can improve brightness levels in the dark areas and provide good contrast in the bright areas. Fig.8 shows the luminance distribution histograms of the image before and after correction. In Fig.8 (b), the red, cyan, blue, mauve and green areas describe the intensity distributions of the input and corrected images of CLAHE, MSRCR, AGC and the proposed method, respectively. As seen in the first row, it is noted that the brightness levels of the crop image obtained in the low-light situation can be improved by the four correction methods. However, the enhancement effect of CLAHE on the low brightness area is insufficient. The performance of MSRCR and AGC is not as good as our method, where the MARCR and AGC methods increase the pixel value of the peak point from 21 to 45 and 94, respectively, and the proposed algorithm improves it to 116. For the crop image obtained in the reflective or exposed scene (see the second and third rows), although the brightness levels of the bright regions can be adjusted, the CLAHE, MSRCA and AGC algorithms almost neglect the correction of the highlight pixels with intensity 255. For the proposed method, we utilize luminance partitioning for obtaining dark and bright regions of the image and built two appropriate Gamma functions to separately deal with luminance correction in the dark and bright regions, respectively. Clearly, the proposed method can be more effectively adjust the dynamic range of the dark and bright regions of the input image.
Moreover, we also perform quantitative measurements to compare the luminance correction performance of the different algorithms. Here, we use image contrast and entropy for image correction quality assessment. The image contrast is evaluated according to the standard deviation of the luminance component. The image with high contrast leads to a high standard deviation value and can provide clearer image details compared with the low contrast image. The image entropy is calculated by the information entropy. We perform evaluation on 160 crop images obtained from low, middle and high natural illumination conditions for the above four correction approaches. The overall performance in terms of image contrast and entropy is given in Table 1. Compared with the other three algorithms, the proposed method produces higher values in terms of the advantage standard deviation and information entropy of the corrected images which indicates that it is effective for image contrast enhancement and providing more meaningful information. Hence, the proposed method can effectively correct image luminance and improve the visual effect of the images.

B. COMPARISON OF CROP SEGMENTATION PERFORMANCE
In order to evaluate the performance of the proposed method on crop segmentation in the images corrected by the proposed correction algorithm, we compare it with the Otsu method  used to segment crops in the images before and after correction. Fig. 9 shows the segmentation results of the three test images obtained via the Otsu and the proposed methods. The original test images taken in different weather (sunny and cloudy) and field (water and soil) environments are shown in the Fig.9 (a). Fig.9 (b)-(d) show the segmentation results by the Otsu method applied in the original image, the Otsu method used in the corrected image, and the improved Otsu in the corrected image, respectively, where background pixels are marked as black. As seen in the Fig.9, without luminance correction in the test images, the Otsu method is greatly affected by the illumination conditions where much misclassification is in the crop segmentation results of the Otsu method, which fails to provide reliable crop information. Although the Otsu method performs well in the corrected image compared with the case of before correction, it still brings over-or under-segmentation of the results in the case of the unbalance distribution. Clearly, the proposed method can obtain accurate segmentation results which can provide reliable crop information for the subsequent extraction of the navigation line.
Additionally, the segmentation performance of the algorithm is evaluated quantitatively by comparing the segmentation results with the reference images labeled manually. VOLUME 8, 2020 The average segmentation accuracy, over-segmentation rate and under-segmentation rate of image segmentation are computed for the above-mentioned 160 crop images. The results are presented in Table 2, where A_Seg, U_Seg and O_Seg represent the segmentation accuracy, over-segmentation rate and under-segmentation rate, respectively. The three quantitative indexes are calculated as follows: where, Bseg is the binary image obtained from segmentation algorithm and B truth is the ground truth labeled manually. B seg ∩ B truth is the number of pixels that are correctly segmented. B Useg represents the number of pixels that should be included in the segmentation result, but are actually not in the segmentation result. B Oseg is the number of pixels that should not be included in the segmentation result, but are actually in the segmentation result. From Table 2, it can be seen that the Otsu method applied in the original and corrected images usually brings a high over-segmentation and under-segmentation rates, respectively. However, the segmentation performance of the proposed method improves that of the comparison algorithms and offers a higher average accuracy of 92.64%. Therefore, the proposed algorithm has good segmentation performance which will greatly increase accuracy of crop row detection.

V. CONCLUSION
This article has presented an automatic crop segmentation based on luminance correction and adaptive thresholding. Based on the sensitivity of the HSV color space to the illumination, we first obtained the luminance of the input image and performed luminance partitioning to create dark, bright and normal areas according to the brightness levels of the image. We then built a partition adaptive Gamma correction function to solve the problem of poor adaptability of the global correction method, especially for the highlight pixels with a pixel value of 255, and segmented plant/background in the corrected image. In the segmentation processing, we used the ExG color index and the Otsu method for pre-segmentation and calculated the threshold for the application of the final threshold-based segmentation. Experimental results demonstrated the proposed algorithm can achieve better performance than the comparison methods which can provide good contrast and preserve more image details. Moreover, the proposed method can achieve better segmentation performance than a single threshold value from the Otsu method which can provide reliable crop information for automated agricultural machinery.