Performance Evaluation of Crop Segmentation Algorithms

This paper presents a thorough evaluation of twenty-one state-of-the-art widely-used crop segmentation algorithms, motived by their significance in vision tasks for further analysis. An ideal crop segmentation algorithm can effectively extract crop information, thus providing an important precondition for the application of intelligent agriculture analytics. In order to enable researchers in this field to fully understand various crop segmentation methods, this paper proposes a new classification strategy of object segmentation by dividing the algorithms into pixel-based and region-based approaches at first, and then systematically evaluating various crop segmentation methods with a unified data benchmark and four common metrics. A new dataset which incorporates crop variety, environment condition and observation distance into consideration is constructed for demonstrating the experiments and comparisons. The effectiveness and robustness of these algorithms were evaluated by three sets of comparative experiments. Based on the quantitative results, we summarize the advantages and disadvantages of the evaluated algorithms from the segmentation performances with four metric indicators. Furthermore, the discussion and evaluation results will provide great support for precision agriculture analysis.


I. INTRODUCTION
Development of computer vision technologies have been widely used to increase the level of agricultural intelligence. Crop segmentation, an application of image segmentation, plays a fundamental role in agricultural information automation. Many issues, such as crop growth stage prediction [1], crop line detection [2], density estimation [3], [4], cover crop identification [5], leaf disease detection [6], [7] and crop biomass monitoring [8], are highly dependent on the performance of crop segmentation algorithms.
Some researchers have summarized recent crop segmentation algorithms in theory [9]. However, little research has been done to investigate and evaluate the performances of the The associate editor coordinating the review of this manuscript and approving it for publication was Young Jin Chun .
proposed crop segmentation algorithms. The effectivenesses of these algorithms are usually assessed on datasets of a specific crop or environment condition with limited number of nuisances and comparisons. It is therefore difficult for the agrometeorologists and developers to choose a proper algorithm given a specific application.
To this end, we present a comprehensive evaluation of twenty-one state-of-the-art crop segmentation algorithms. This is the first comprehensive evaluation study of crop segmentation algorithms, to the best of our knowledge, which considers both classical and the latest methods with assessment on benchmarks addressing a variety of applications [10]. In this paper, the benchmark dataset is constructed by us. Different from the images captured in the lab or a constrained environment, the images in our proposed dataset are all in the open wild, followed by different crop varieties and observation distances, complicated background and changeable weather, which are all the common problems when segmenting the crops. The terms Intersection over Union(IoU), Sensitivity(Se), Specificity(Sp) and accuracy (Ac) are used to measure the quantitative performance on mentioned situations, ensuring a balanced examination of the results. The contributions of this paper are summarized as: • A new classification strategy is proposed to summarize the twenty-one crop segmentation algorithms in the past two decades.
• A new dataset is provided, which contains four crop varieties (cotton, maize, rice, wheat), seven environment conditions (highlight, sunny, cloudy, overcast, shady, rainy, complex background) and two observations (nearrange and canopy). It is hoped that it can be served as a benchmark for related works.
• Three sets of comparative experiments with different variables (variety, environment, distance) were set up for comparison, and the experimental results were demonstrated with four different evaluation indicators. The remainder of this paper is organized as follows. Sec. II gives a description and analysis of twenty-one state-of-the-art crop segmentation algorithms. Sec. III shows the evaluation methodology, which consists of the dataset, the performance measures and the implementation details of the evaluated algorithms. The experimental results are reported in Sec. IV, while the conclusions are drew in Sec. V.

II. CROP SEGMENTATION ALGORITHMS A. CROP SEGMENTATION ALGORITHMS DESCRIPTION
This subsection briefly reviews twenty-one state-of-the-art crop segmentation algorithms, and describe the theory and computational steps of each algorithm.
• Excess Green Index (ExG) [11]: The ExG method is based on the fact that the value in the green channel is larger than that in other channels. After trying several color vegetation indices, such as r − g, g − b, g−b r−g and 2g − r − b, Ref. [11] found that the Eq. 1 was the best choice for separating plants from bare soil.
where R, G, and B are the chromatic coordinates: where R * , G * and B * are the normalized RGB, in the range of [0, 1]. The ExG index has been widely used and has performed very well in separating plants from non-plants [12]- [14], because ExG has provided a clear contrast between plants and soil with low cost. However, the method does not perform well when the daylight is changeable or the background is complicated.
• Environmentally Adaptive Segmentation Algorithm (EASA) [15]: An environmentally adaptive segmentation algorithm (EASA) was developed for outdoor field plant detection. Based on a partially supervised learning process, the algorithm specific objective was to investigate the feasibility of using a minimally supervised learning procedure for color image segmentation under naturally changing outdoor lighting conditions. The EASA-based algorithm had a much higher segmentation accuracy in poor lighting conditions than the static algorithm. But it is not effective to segment plants at early growing stage where the cotyledons start to appear [16].
• Normalised Difference Index (NDI) [17]: The proposed method uses color information to discriminate vegetation from background. Then shape analysis techniques are applied to distinguish between crop and weeds afterwards. After trying three color indexes on the images, Eq. 3 was demonstrated to be the best. The NDI algorithm is robust for sunny images to some extent, but does not perform well when in highlight or cloudy images.
• Colour Index of Vegetation Extraction (CIVE) [18]: The CIVE method based on a study carried out in soya bean and sugar beet fields has enhanced the green information for separating the green plant portion from the background by Eq. 4.
where R, G, B were the values of the RGB intensity of each pixel. CIVE has better plant segmentation than the Near-infrared (NIR) method because it provides greater emphasis on the green areas. However, the method performs poorly in cloudy, shady or highlight images.
• Vegetative Index (VEG) [19]: Different from above color indice algorithms, the VEG method converts an RGB image to grayscale with the following formula: where the r, g, b are the values of the red, green and blue channels, and the α is a constant parameter with a value of 0.667. It is noticed that the VEG is robust to lighting change, but does not perform well when the background is complicated. In addition, the segmentation results are also strongly affected by the crop variety, because different green crops are grown in different environments.
• Mean-Shift algorithm (MS) [20]: With mean shift and a BPNN(Back Propagation Neural Network), a segmentation algorithm for separating images containing green vegetation was presented. The proposed algorithm mainly consists of feature extraction and image segmentation. Firstly, multiple color features, such as hue VOLUME 8, 2020 and saturation in HSI color space were extracted. Secondly, the image was classified into green and non-green vegetation with the extracted features, using a meanshift segmentation algorithm and a BPNN. Although the error of mis-segmentation is lower, this method has a long execution time, which is not suitable for real-time applications.
• Mean-Shift algorithm with Fisher Linear Discriminant (MS_FLD) [21]: A hybrid method combining mean shift (MS) and Fisher linear discriminant (FLD) is implemented to improve the performance of crop segmentation. A point-line-distance-based strategy for weighting the training data whose distribution is long and narrow is adopted by the MS_FLD. Compared with the MS algorithm [20], the MS_FLD overcomes the negative impact from the shadows. However, the limitation of this algorithm is that it is merely suitable for two-class segmentation issues in theory.
• Relevant Textures (RT) [22]: The authors propose an approach that exploits the performance of existing color index strategies by combining them, which contains ExG, ExGR, CIVE and VEG.
The experimental results show that the segmentation quality can be improved when applying the mentioned color indexes simultaneously. In addition, it can be applied to any type of crops, such as wheat or rye. However, it does not perform well when the light is high or low.
• Linear Color Models (LCM) [23]: Although the objects detected with LCM method are not crops but red peaches in orchard images, the method is also worth being compared. That is because the segmentation is performed by comparing the color distance from each pixel to the different previously defined linear color models (crops, branch, ground, other interferent) in the field. The proximity to the different linear color models is the criteria used to individually classify the pixels of the images, and there is no need to define additional threshold or radial proximity distances in the linear color models. This method shows good adaptability in outdoor environments, but it is easy to divide the shadow pixels into plants (over-segmentation).
• CIEL*a*b color space and segmentation (LabSeg) [24]: A new morphology modeling method is utilized to establish a crop color model in the CIE L*a*b* (or Lab for simplification) color space for crop image segmentation. The proposed segmentation method contains a supervised off-line learning stage and an online segmentation stage. In the supervised learning stage, morphology modeling is applied to deal with the color characteristics of the crop with respect to the pixel lightness component and establish the crop color model. The method is robust to the variation of illumination in the field. The experimental results in reference [24] demonstrate that using different types of structuring elements for morphological modeling will not improve the segmentation performance of the proposed method significantly.
• Expert System (ES) [25]: The ES algorithm consists of two main modules: (1) decision making, based on image histogram analysis and (2) greenness identification, where two different strategies are proposed, the first based on classical greenness identification methods and the second inspired by a Fuzzy Clustering approach. The incoming image is transferred to the decision making module, where based on histogram analysis a decision is made according to the contrast of the histogram. If there is sufficient contrast, they apply the combined method [22]. Otherwise they apply a down-sampling together with smoothing for image preprocessing. After this a procedure inspired by the FC approach allows a threshold to be determined. The binary image is obtained through a simple decision rule. But under adverse environmental conditions most of them fail or do not work properly.
• Affinity Propagation-Hue Intensity (AP_HI) [26]: Facing the problem of complex in-field crop segmentation, a novel crop segmentation method (AP_HI) that combines a hue-intensity (HI) look-up table and affinity propagation (AP) clustering algorithm [27] is proposed. It can produce a great segmentation result from a small amount of training samples while meeting the requirements of actual field environment. Additionally, it is robust and not sensitive to the challenging variation of outdoor luminosity and complex environmental elements. However, for an intercropping pattern, mutual occlusion between two crops will have negative impact.
• Decision Tree based Segmentation Model (DTSM) [28]: The method is based on decision tree created by CART algorithm and image noise reduction filters. Using the R, G, B color information of each pixel in the images, 18 color features (r, g, b; Y, Cb, Cr; H, S, L; H, S, V; L*, a*, b*; L*, u*, v*.) defined in 6 ordinarily used color spaces (rgb, YCbCr, HSL, HSV, CIEL*a*b* and CLEL*u*v*) were derived. The selection of color features for extracting vegetation is done using a feature selection technique known as wrapper [29]. The feature subset with the highest evaluation is chosen as the final set on which to run the learning algorithm. The decision tree is evaluated using the selected color features of the training dataset by using the CART classifier [30]. Then, the constructed tree model is applied to conduct segmentation on test images, resulting in each pixel belonging to either vegetation or background class, which finally generated binary images of vegetation. The method performs better than other methods when the crops appear in strongly shadowed and specularly reflected parts. Nevertheless, the robustness of such model was only tested on wheat images, more crops and more complicated environment should be considered afterwards.
• Mean-Shift with Colour Index of Vegetation Extraction (MS_CIVE) [31] Mean-Shift with Excess Green Index (MS_EXG) [31] Mean-Shift with Visual Vegetation Index (MS_VVI) [31]: They investigates the combination of vegetation indices (color index of vegetation extraction(CIVE), visual vegetation index(VVI), and excess green(ExG)) and the mean-shift algorithm, based on the local density estimation in the color space on images acquired by a low-cost system. The objective is to detect green coverage, gaps, and degraded areas. Combining local density estimation and vegetation indices improves the segmentation accuracy when compared with the competing methods. It deals well with the images in different conditions and with regions of imbalanced sizes, confirming the practical application of the low-cost system. The idea is to take advantage of the local maxima generated by the MS, which removes texture and small irregularities of the image and afterward extracts the vegetation index. Among the investigated indices, the ExG index appears to be the best choice when only the visible band is available, since CIVE and VVI produced lower accuracies and unstable results when dealing with different acquired regions of the crop, showing higher variance in the presence of outliers. When computed over the previously segmented image, both CIVE and ExG showed good results, without outlier results. The VVI, however, could not be improved by using a mean-shift pre-segmentation. • Denoising Autoencoder (DA) [33]: A novel strategy based on the deep learning is utilized to establish the crop classifier in the RGB vector color space in order to realize the specific crop image segmentation. The proposed crop segmentation method can not only segment green plants in the field, but segment specific crop like cotton. The proposed method for separating green plants into four classes in the images had the highest precision without applying any noise reduction technique. Although the correct classification rate will be further improved after applying noise reduction technique, it is not very stable in cases who are the image acquisition service moves, on account of various forms of noises.
• Discrete Wavelet Transform (DWT) [34]: They propose a segmentation strategy for agricultural images on basis of the discrete wavelet transform. To combine greenness and spatial texture information together, they make use of vegetation indices and the wavelets transform. The irregular spatial texture distribution, existing in soil and green parts (crops and weeds), is captured appropriately through the wavelets decomposition in horizontal, vertical and diagonal detail coefficients, which conveniently combined together with the approximation coefficients provided by the wavelets transformation allows important improvements in image segmentation for the intended objective.
• MRFMAP framework (PFMRF) [35]: They propose a novel crop extraction method resistant to the strong illumination using probabilistic superpixel Markov random fields. The method is based on the assumption that color changes gradually between highlight areas and its neighboring non-highlight areas and the same holds true for the other regions. This priori knowledge is embedded into the MRF-MAP framework by modeling the local and mutual evidences of nodes. Superpixel and fisher linear discriminant are utilized to construct the probabilistic superpixel patches. The loopy belief propagation algorithm is adopted in the optimization step and the label for the crop segmentation is provided in the final iteration result. The method is robust to the variation of illumination and also capable of extracting the crop from the shadow regions.
• Joint Crop Tassel segmentation (JOINT_CT) [36]: They proposed a region-based approach for joint crop segmentation. First, an efficient graph-based segmentation algorithm and simple linear iterative clustering (SLIC) are leveraged to generate region proposals. Then colors are evaluated with ensemble neural networks specific to each intensity, aiming to achieve robustness to illumination. In addition, two simple but effective strategies are devised to accelerate the color statistics extraction and ensemble model prediction. The effectiveness and efficiency of the method are demonstrated on the two segmentation tasks, respectively. The method significantly outperforms other state-of-the-art approaches on tassel segmentation, but the performance on other crops have not been demonstrated yet.
• CIE LUV color space and Support Vector Machines (LUV_SVM) [37]: They propose a method which uses central pixel neighborhood information to classify a pixel through support vector machines (SVM). The underlying idea is to include contextualized information on the learning system. Additionally, they compare VOLUME 8, 2020 the results with threshold (including color index approaches) and learning based approaches, performing experiments using three datasets: tomato, maize [38] and carrot [39]. They provide a new dataset containing crops of tomato with high variability, complex background, and accurate manual annotations. It was demonstrated that machine learning methods have a more robust behavior than the ones based on thresholding, which have difficulty segmenting under varying illumination conditions. These difficulties were improved by learning based approaches in the most cases. Nonetheless, images with back light are still challenging.

B. CROP SEGMENTATION ALGORITHMS CLASSIFICATION
With the development of computer vision technologies, more and more crop segmentation algorithms are proposed nowadays, and researchers have classified those algorithms accordingly. In Ref. [35], the crop segmentation algorithms fell into two broad categories, color-index-based and learning based approaches. In Ref. [38], Lu et.al divided the crop segmentation algorithms into three categories, including the color index based, the learning based and the color model based approaches. Ref. [9] also proposed a classification strategy, they divided the algorithms into color index-based, threshold-based and learning-based approaches. Although above classification methods can distinguish and define the crop segmentation algorithms to some extent, they cannot cover all the proposed algorithms. Moreover, the boundaries between two categories are not clear enough. As a result, we propose a new classification strategy by dividing the algorithms into pixel-based and region-based approaches.

1) PIXEL-BASED ALGORITHMS
For the pixel-based algorithms, color is the unique cue in segmentation due to the limited information of one pixel, so that various color spaces and classifiers are used to solve crop segmentation issues. As shown in Fig.1, we refine the classification according to the feature extraction method, such as color indice, learning and wavelet. Among them, color-indice-based algorithms contain two subcategories, including single and combined indices. The ExG [11], NDI [17], CIVE [18], and VEG [19] algorithms belong to the single class, they make use of single color indice for crop segmentation. Conversely, the RT [22] and ES [25] algorithms segment the crop images with combined color indices.
In addition, the learning algorithms can be divided into three subcategories: unsupervised, semi-supervised, and supervised. The aforementioned ES [25] algorithm belongs to an unsupervised learning algorithm. The EASA [15] algorithm is a semi-supervised learning algorithm. Finally, we divide supervised learning into four categories: color model, decision tree model, neural network model and SVM model. The Labseg [24], AP_HI [26] and LCM [23] algorithms are color model algorithms. The DTSM [28] algorithm belongs to the decision tree model algorithm. DA [33] belongs to the neural network model algorithm. The LUV_SVM [37] algorithm belongs to the SVM model algorithm. In addition to the common color indice and learning algorithms, some people use the wavelet method for crop image segmentation, such as the DWT [34] algorithm.

2) REGION-BASED ALGORITHMS
The pixel-based approaches with its merits of simplicity and fast processing speed, usually need to directly feed the pixellevel feature to the model. Indeed, this line of research has achieved a moderate degree of success. However, some drawbacks are appeared. One is that it may produce noise with a high degree of probability, since the model has to visit all the pixels in an image. The other is that the edge of segmented regions may not be well preserved, because existing methods are not intentionally designed for edge preservation. That is to say, integrity of agriculture products cannot be guaranteed and more noise points are taken in. Therefore, for the product which has relatively regular shape, salient color and consistent texture, some researchers focus on region-based methods.
The fundamental step of region-based crop segmentation algorithms is to divide the image into non-overlapping regions, which are supporting regions for feature vectors and primitives to reduce computational complexity. Then they can extract the features from generated regions afterwards. Fig.2 show the classification strategy of region-based approaches. As shown in Fig.2, we refine the region-based algorithms with meanshift-based, SLIC and graph-based, on basis of region generation mode. The meanshift-based algorithms include color index and learning-based. The MS [20], MS_CIVE [31], MS_EXG [31] and MS_VVI [31] algorithms belong to the color index class, the MS_FLD [21] algorithm belongs to the learning class, and the MS_FLD [21] algorithm is a supervised learning algorithm. The PFMRF [35] algorithm is a graphic model algorithm under the SLIC category. The JOINT_CT [36] algorithm is a learning algorithm based on the neural network color model in the SLIC+graph-based category.

C. CROP SEGMENTATION ALGORITHMS ANALYSIS
Color is a vital feature which affects the performance of crop segmentations strongly. After reviewing all the crop segmentation algorithms in the recent two decades, we observe that almost all the crop segmentation algorithms use single or multiple color spaces. Due to the complex outdoor environment, the in-field crop image is easily affected by different illuminations, thus the color cannot be the only impact factor for crop segmentation. Therefore, more researchers choose to expand the capability of feature description by using texture or semantic features.
Through the description of the above image segmentation algorithm, we found that in addition to the RGB color space model commonly used in computer technology, many algorithms use other color space models, which we have been summarized. Fig.3 shows different color space models used by pixel-based classification and region-based classification image segmentation algorithms. The main problem explained by Fig.3 is that the pixel-based classification method relies too much on color information, and the accuracy of the segmentation algorithm has different performance in different color spaces; the region-based classification method is preconsidered in region-generation. Due to the influence of environmental factors, pixels with the same or similar attributes are assembled, so the region-based classification method is less sensitive to color space.

III. EVALUATION METHODOLOGY
All the algorithms presented in Sec. II have been evaluated on the proposed dataset, where different factors of crop varieties, environment conditions and observation distances are taken into consideration. The performances of different crop segmentation algorithms are measured with four indicators, which are widely used for evaluation. This section also presents the implementation details as well.

A. DATASET
The in-field crop images were collected by the ground-based non-contact observation system, described in Refs. [40], [41]. With this system, continuous and multi-perspective observations can be realized. In view of crop variety, environmental conditions and observation distance, a total of 340 images in all were selected to construct a high performance imaging dataset, with a data size of approximately 2 GB. Each image in the dataset has a corresponding ground-truth image with a high degree of accuracy. They are all manually annotated using Adobe Photoshop CS5 software. Due to gene expression, different variety of crops appear multiple forms in color, shape and other characteristics. To evaluate the robustness of different crop segmentation algorithms, we take crop variety into account when constructing the dataset. Specifically, the dataset consists of four crops in total (Fig. 4), including cotton, maize, rice and wheat. Each of them incorporates 86, 190, 56 and 8 images, respectively. Considering different environmental conditions they may appear in the wild influenced by weather and planting characteristics, the dataset can be split into seven common scenarios, including highlight, sunny, cloudy, overcast, shady, rainy and complex background, as illustrated in Fig. 5. Each dataset contains 39, 126, 70, 61, 18, 15 and 11 images, respectively. In addition, the distance between crop and the observation system also affects the accuracy of crop segmentation. Therefore, as shown in Fig. 6, the dataset can be divided into 261 nearrange and 79 canopy images in terms of observation distance.
Consequently, we utilize our constructed dataset to build the training and test set for crop variety, environment condition and observation distance, respectively. Under such a distribution, the robustness evaluation of either segmentation algorithm will be more convincing. We believe this dataset can be of great value for computer vision applications in agriculture. It provides unified platform for the comparison of different methods.

B. CRITERIA
The performance of segmentation algorithms cannot be fully described by a single evaluation indicator. To quantitatively analyze the performance of above methods, four standard evaluation metrics are introduced for assessment. Intersection over Union (IoU) is a popular measure for semantic segmentation of objects, because it penalizes both over-segmentation and under-segmentation [42]. In addition, Sensitivity (Se), Specificity (Sp) and Accuracy (Ac) are commonly used for performance evaluation in classification problems [43]. Se is thus a measure of accuracy of the classifier of true cases of specific crop. Sp is a measure of accuracy of the classifier of false cases of specific crop. Ac is the number of correct classifications divided by the total number of classifications. The error rate is one minus Ac. Formally, these measures take the form as where TP is the number of pixels predicted as object when these pixels are actually objects. TN is the number of pixels  predicted as background when these pixels are actually background. FP is the number of pixels predicted as object when these pixels are actually background. And FN is number of pixels predicted as background when these pixels are actually objects. Mean valuesμ and standard deviations σ are employed to measure effectiveness and robustness with evaluation metrics for different approaches.

IV. RESULTS AND DISCUSSION
Based on machine learning, the accuracy of a classifier strongly depends on the selection of training data [28]. First, crop variety, environment condition and observation distance are three vital elements in dataset selection for crop segmentation. Then, for test and display purposes, the canopy images are rescaled from 2736 × 3648 to 912 × 1216, The scale of near-range images are set to either 500 × 500 or 601 × 601, which are all randomly selected from the original canopy images. Finally, the quantity ratio of training to test is set is 7:3 for each experiment. Additionally, both the training and the testing sets for the cotton segmentation are randomly selected from the constructed dataset. In order to evaluate the performances of different crop segmentation algorithms, we compare 21 state-of-the-art algorithms in total, and a general method with sufficient flexibility and scalability for certain application is found therein. We have divided the datasets and set up three sets of comparative experiments of different variables: (1) crop varieties; (2) environment conditions; (3) near-range and canopy images. The specific experimental results are as follows:

A. PERFORMANCES ON DIFFERENT CROP VARIETIES
Crop variety is an important factor affecting the accuracy of crop image segmentation. Different crop varieties tend to exhibit different brightness, color and texture characteristics. In this experiment, we take four different crops (cotton, maize, rice and wheat) into consideration. In terms of the criteria of control variate method, those crop images with different varieties were compared in twenty-one crop segmentation VOLUME 8, 2020  Through the visualization results and quantifiable outcomes of different crop segmentation algorithms in four kinds of crop varieties, we observe that: • With regard to the IoU metric, most algorithms perform better on cotton than other crops, with general mean values of more than 60%. As for the rice images, the mean values ofμ are lower on IoU, but the average mean valuesμ on AC are higher than other crops, and the   standard deviation σ is also low on AC. It is worth noting that most algorithms have rather higher segmentation performance in rice crop images than other crops.     while in other crops the mean valuesμ on AC is only 85% or less; while the mean valueμ of the DWT algorithm is only 67.94% on AC for wheat images, but more than 92% on other crops.
• Labseg, CIVE and NDI algorithms achieve high mean valuesμ on AC in each crop image, and the performance is the most stable. The average mean valuesμ of CIVE algorithm can reach 94.56% on AC in rice images.
From the above results, it is not difficult to conclude that some crop image segmentation algorithms are designed for specific crops, and often do not achieve good segmentation in other crops, but similar effects can be obtained in crops with similar texture. If image segmentation is performed for a specific crop, a specific algorithm can be selected. If image segmentation is performed for multiple crops, a higher comprehensive performance algorithm is required.

B. PERFORMANCES ON DIFFERENT ENVIRONMENT CONDITIONS
Changeable weather is an important factor affecting the accuracy of crop segmentation algorithms. Different weather conditions will produce different light intensity, humidity, and illumination angles, all of which will affect crop segmentation to varying degrees. In this experiment, we used the weather as a variable to classify the dataset, providing seven different weather environment images for highlight, sunny, cloudy, overcast, shady, rainy and complex background. These images were compared by twenty-one crop image segmentation algorithms. Fig.11-17 show the results of the image segmentation algorithms in different weather environments. Tables 5-11 show the mean values and standard deviations of four metrics on forward and downward images. These are highlight, sunny, cloudy, overcast, shady, rainy and  From the results of the seven environment images in different crop segmentation algorithms and the data from four assessment indicators, we observe that: • In overcast, shady and rainy images, the mean valuesμ on AC and IoU are lower, most of them are less than 70% and 57.49%, respectively. In highlight or sunny images, the segmentation effect of each algorithm is uneven, and the mean valuesμ on IoU is the lowest, with the value of 21.28%. In addition, the Labseg algorithm has the highest value 97.12% on AC on the cloudy scenario, while the performance in highlight images is not ideal  • Some algorithms do not perform well when the light intensity is too high or too low, because the illumination   enhancement will cause a line shadow on crop plants. For example, the ExG algorithm performs worst when the light intensity is too high or too low. It is lower than the second worst LCM algorithm by 15.6% and 5.56% lower than the second worst algorithm LCM on IoU in sunny and rainy images, respectively. Therefore, some  algorithms have low sensitivity to background errors and changes in illumination intensity, and do not exhibit good adaptability in outdoor environments. It is not difficult to conclude that the crop segmentation results conducted by many algorithms are strongly affected by the illumination. In addition, complex background will bring more complicated texture features, which makes in-field crop segmentation more difficult. Moreover, the Labseg and DA algorithms outperforms other state-ofthe-arts on different environment conditions.

C. PERFORMANCES ON NEAR-RANGE AND CANOPY IMAGES
In this subsection, we use observation distance as a variable, conducting experiments on near-range and canopy images with twenty-one crop segmentation algorithms, as illustrated in Fig.18 and Fig.19. Table.12 and Table.13 show the mean values and standard deviations of four metrics on near-range and canopy images. The higher the values of above evaluation metrics are, the better the performance of the segmentation algorithm. From Table.12 and Table.13, we can observe that the highest performances of 88.49%, 93.06%, 94.57% and  94.82% with the lowest standard deviations of 6.13%, 5.81%, 2.24% and 3.27% on IoU , Se, Sp and Ac, respectively.
From the visualized results of near-range and canopy images in different crop segmentation algorithms and the quantitative data with four metrics, we observe that: • The mean valueμ in near-range images is higher than that in canopy images on AC, and lower than that on Sp. Among them, the DWT and MS_FLD algorithms are superior to other algorithms in general.
• The values of AP_HI, MS_VVI, ExG and LCM algorithms in canopy images are higher than that in nearrange images. Among them, the AP_HI algorithm outperforms other algorithms with the values of 89.22% and 78.07% on AC in canopy and near-range images, respectively. On the contrary, the RT, MS and MS_CIVE algorithms work well in near-range images. Among them,

V. CONCLUSION
This paper has presented a thorough evaluation of twentyone state-of-the-art crop segmentation algorithms on our constructed dataset. In addition, this paper proposes a new classification rule based on the segmentation objects, dividing those algorithms into pixel-based and region-based categories. Our goal is to find a universal approach which adapts to a variable environment with sufficient flexibility, stability and scalability. To this end, we have constructed a dataset, which takes crop variety, environment condition and observation distance into consideration.
In three sets of comparative experiments which considers different crop varieties, environmental conditions and observation distances, we selected appropriate parameters and evaluated the effectiveness and robustness of different crop segmentation algorithms with four common segmentation evaluation indicators. In light of these evaluation outcomes, we summarize the findings of this paper into several points as follows: • The MS_FLD and DWT algorithms are more stable and most effective among them. Most other algorithms can only segment a particular crop or work in a specific kind of environment condition. No algorithm has the highest performance and the lowest standard deviation in the comparison experiments of evaluation indicators. That is, no algorithm can be applied to all crop varieties, environmental conditions and any observation distance. We can only choose a more robust algorithm among them, or select a specific algorithm for specific conditions.
• Although some algorithms achieved good segmentation results, there are still many problems to be solved: (1) Different illumination conditions (cloudy, overcast, and highlight, etc.) will affect the crop segmentation quality, resulting in high false rejection rate. Specifically, all the algorithms cannot achieve ideal segmentation performances for overcast images; (2) Complex background environments resulting from straws, stones, soil, water pipes and other residues will decrease the accuracy of crop segmentation algorithms.
• After reviewing the aforementioned crop segmentation algorithms, we realized the opportunities and challenges in the field of crop segmentation: a new segmentation algorithm which adapts to different illumination conditions and complex background environment is highly needed.
Therefore, this paper can be treated as a useful reference for the selection of crop segmentation algorithms with respect to different applications. Due to the complexity of crop segmentation, researchers need to make unremitting efforts to solve this classical problem. We hope our research can provide the necessary basis for future related work.
ZIYUN HUANG received the B.S. degree from Dalian University, Dalian, China, in 2017. She is currently pursuing the M.S. degree with the School of Computer Science and Engineering, Wuhan Institute of Technology, Wuhan, China.
Her current research interests include computer vision and machine learning, with particular emphasis on image processing, object detection, and various computer vision applications in agriculture.
ZHIGUO CAO received the B.S. and M.S. degrees in communication and information system from the University of Electronic Science and Technology of China, Chengdu, China, and the Ph.D. degree in pattern recognition and intelligent system from the Huazhong University of Science and Technology, Wuhan, China.
He is currently a Professor with the School of Automation, Huazhong University of Science and Technology. He has authored dozens of articles at international journals and conferences, which have been applied to automatic observation system for object recognition in video surveillance system, for crop growth in agricultural and for weather phenomenon in meteorology based on computer vision. His research interests spread across image understanding and analysis, depth information extraction and object detection. His projects have received provincial or ministerial level awards of Science and Technology Progress in China.
HAO LU received the Ph.D. degree from the Huazhong University of Science and Technology, Wuhan, China, in 2018.
He is currently a Postdoctoral Fellow with the School of Computer Science, The University of Adelaide. His research interests include computer vision, image processing, and machine learning. He has worked on topics including visual domain adaptation, fine-grained visual categorization, and miscellaneous computer vision applications in agriculture. His current interests are object counting and dense labeling problems, such as low-level image matting, image denoising and high-level semantic segmentation, and depth estimation.
HAIHUI WANG received the Ph.D. degree in pattern recognition and intelligent system from the Institute of Image Recognition and Artificial Intelligence, Huazhong University of Science and Technology, Wuhan, China.
He is currently a Professor with the School of Computer Science and Engineering, Wuhan Institute of Technology. He has authored dozens of articles at international journals and conferences, which have been applied to image recognition and moving object detection based on computer vision. His research interests spread across digital image processing, computer vision, artificial intelligence theory and application, and intelligent computing.
SHUIPING ZHANG was born in Hubei, China, in 1988. He received the Ph.D. degree from the School of Automation, Huazhong University of Science and Technology. He is currently a Lecturer with the School of Computer Science and Engineering, Wuhan Institute of Technology, Wuhan, China. His research interests include image processing and hardware implementation. VOLUME 8, 2020