Green Plums Surface Defect Detection Based on Deep Learning Methods

Green plums are a characteristic fruit resource in China, with a long history of cultivation. Many surface defects will appear in the growth, transportation and preservation of green plums which seriously affect the processing quality of by-products. The existing manual sorting method of green plums is limited by the experience of workers. It is difficult to ensure the quality and speed of detection. Therefore, the formation of automatic detection of green plums surface defects is of great significance to the development of green plum industry. According to the surface defects of green plums, this paper divides green plums into five categories: rot, cracks, scars, rain spots and normal. A total of 1235 images of green plums were obtained by self-built image acquisition device. The WideResNet50-AdamW-Wce model based on WideResNet model was built to classify the surface defects of green plums. Accuracy, recall and F1-measure were selected as the indexes to evaluate the accuracy of classification. The accuracy of classification reached 98.95%, and the classification accuracy of rain spots, normal, scars, rot and crack reached 100%, 99.56%, 98.59%, 98.25% and 96.10% respectively. Comparing the performance of ResNet50-SGD, WideResNet50-SGD, WideResNet50-SGD-Wce, and WideResNet50-AdamW network models, the F1-Measure based on WideResNet50-AdamW-Wce is the highest in each defect, and more greengage defect features can be learned. The detection results can meet the production needs of plum deep processing enterprises – evaluating 1800 green plums per hour on the assembly line.

China, Japan and Southeast Asia [1]. Green plum fruit is 23 rich in vitamins, trace elements, amino acids that are bene-24 ficial to human protein composition and metabolic functions, 25 and a variety of high-quality organic acids [2]. Since the 26 early 1990s, with the increasing demand for green plums 27 in Japan and South Korea, a large number of fresh green 28 plums have been exported, and the output of green plums in 29 China has also increased year by year [3]. Green plums are 30 often sold to the market as processed products. Green plum 31 processed products mainly include green plum wine, green 32 The associate editor coordinating the review of this manuscript and approving it for publication was Jad Nasreddine . plum juice, candied fruit, etc. At the same time, the newly 33 developed green plum health care products such as green 34 plum health lozenges and enzyme plums are also favored by 35 consumers [4]. 36 With the promotion of automatic picking technology, 37 mechanical harvesting can ensure the harvesting efficiency 38 of green plums, but the defective green plums mixed in after 39 mechanical harvesting, as well as the defective green plums 40 that are bumped and rotted during transportation and storage 41 will affect the product quality. The existence of defects will 42 seriously affect the product quality [5]. Before deep process-43 ing of green plums, these defective fruits need to be removed 44 or graded in time. Defect classification refers to the process of 45 identifying and processing the surface defect images of green 46 plums. At present, the defect classification of green plums 47 is still dominated by manual detection. This process mainly 48 relies on the experience of operators, so it is highly subjective 49 and inefficient. In addition, long-term operation is prone to 50 to measure the optical coefficient of peach after injury at 107 different maturity levels. The optical properties showed that 108 tissue damage was earlier than the appearance observed 109 by naked eyes. Huang et al. [16] selected three effective 110 bands for detecting apple damage based on hyperspectral 111 imaging technology. The static and online detection results 112 for minor damage were 91.5% and 74.6%, respectively, 113 realizing the transformation from the hyperspectral research 114 level to the multispectral application level. Unay et al. [17] 115 built a multispectral imaging system for two-color apple 116 grading, and proposed a two-class grading scheme with 117 an overall detection accuracy of 93.5%. Yang et al. [18] 118 made a comparison between the performance of using image 119 classification networks(GoogLeNet and VGGNet) and object 120 detection networks((Faster R-CNN and YOLOv3) to detect 121 broadleaves and grasses. The research found that the use 122 of VGGNet as the decision-making system for the machine 123 vision subsystem seems to be a viable option for the precise 124 spraying of herbicides in alfalfa.

125
Domestic and foreign related technical personnel have 126 more research on defect detection algorithm, according to 127 the use of different algorithms will be roughly divided into 128 four categories: defect detection based on statistics, based on 129 frequency domain, based on model, and based on learning, 130 their advantages and disadvantages are shown in Table 1.

131
Green plums are not big but the samples are numer-132 ous, it remains high similarity between different defects on 133 the surface, increasing the difficulty to distinguish varies 134 defects. Traditional machine vision technology requires man-135 ual extraction of defect features, which easily leads to incom-136 plete features and affects detection results. The deep learning 137 technology overcomes the shortcomings of manual feature 138 extraction. Deep learning networks can automatically extract 139 defect features from datasets through convolutional layers. 140 In order to meet the needs of some domestic green plums 141 products processing enterprise(Nanjing Longlijia Agricul-142 tural Development Co., Ltd. ( China )) to dynamically detect 143 1800 plums per hour, in this paper, the improved WideResNet 144 deep learning model was combined with machine vision 145 technology to perform a multi-index classification of green 146 plum surface defects.

147
The contribution of this paper are: a) multi-index classifi-148 cation of green plum surface defects; b) a self-designed green 149 plum surface image acquisition device; c) the application of 150 AdamW optimizer and Weighted cross entropy (Wce) loss 151 function in the green plum defect detection network based 152 on WideResNet network, to ensure the network can accu-153 rately learn the defect features of green plums and effectively 154 improve the classification performance of the networks.

156
A. IMAGING      representations, giving close to perfect illumination invari-205 ance and very good performance across a change in device.

206
In this paper, the original images were Gaussian filtered with  Table 2. networks, because a deeper network can enhance the feature 227 fitting ability of the network, and a thinner network can 228 reduce the amount of parameters of the network, reducing 229 the computational cost. However, Sergey et al. [20] did the 230 opposite and proposed Wide ResNet in 2017. As the name 231 suggests, this network structure widens the number of chan-232 nels of ResNet, and breaks the mainstream neural network 233 design concept.

234
Although ResNet can be used to alleviate gradient vanish-235 ing by adding skip connections, it is still possible that gra-236 dients only pass through skip layers during backpropagation 237 without updating parameters. In order to solve this problem, 238 the structure shown in Figure 3 is used to widen the number 239 of channels of each convolutional layer, and reduce the depth 240 of the network. In addition, due to the increase in the number 241 of channels, redundant feature maps will inevitably appear in  (1) and (2) where g t is the gradient, m t is the first moment of the gradient, is corrected, as shown in formulas (3) and (4): The formula for AdamW parameter update is shown in 287 formula (6):

289
where θ is the parameter, η is the learning rate, α value is 290 0.001, ζ value is 10 −8 , and ω is a real number.

291
Since AdamW optimization algorithm adds a regular term 292 to the loss function of the Adam optimization algorithm, the 293 overall convergence speed of the model is faster, and the 294 overfitting is reduced. AdamW algorithm can also correct 295 slow convergence speed, large loss function fluctuation and 296 disappearance of learning rate in other optimization algo-297 rithms. Each sample will output an N-dimensional array through the 301 softmax layer after the feature extraction of the complex net-302 work. Each dimension in the array corresponds to a different 303 defect category, that is, the N classification problem, which 304 can be expressed as formula (7): where O is the final output of the network; P (·) is the match-308 ing probability between the output result and the correspond-309 ing category; n is the different defect categories; W n and b n 310 are the weight matrix and bias of each category respectively; 311 exp (·) is the exponential function.

312
The network uses the cross-entropy loss function to deter-313 mine the gap between the actual output and the expectation, 314 giving each sample an equivalent error loss. However, in the 315 actual classification of green plum defects, different defects 316 show an unbalanced distribution. In order to obtain excel-317 lent classification performance, the neural network model 318 often ignores samples of few categories and focuses on most 319 categories, resulting in high overall accuracy but the low 320 accuracy of few categories samples. Therefore, in order to 321 solve the difficulty of classification of unbalanced sample 322 defect categories, the Weighted cross entropy Loss(WceLoss) 323 function adds corresponding penalty coefficients to the error 324 losses of different categories on the basis of the original loss 325 function, and realizes the weighted average loss of different 326 categories of errors. The penalty coefficient can be set accord-327 ing to the balance between different categories, which can be 328 expressed as: 329 c n = mean {a n } N n=1 a n (8) 330 where c n is the penalty coefficient; a n is the number of 332 different categories of samples; mean {·} N n=1 represents the 333 median of the number of N kinds of classification samples. 334 Q is the total number of samples; 1 (·) is the discriminant 335 function, which is 1 when the parentheses are established, and 336 VOLUME 10, 2022   (10):  of samples (TP+TN+FP+FN). That is, the proportion of the 380 correct samples for all five types of defects to be predicted in 381 the total samples.

382
The test set green plum data was imported into the trained 383 green plum defect classification network model, and the test 384 results were obtained after 30 trails, as shown in Table 4.

385
It can be seen that the accuracy rate of the 386 WideResNet50-AdamW-Wce network for the classification 387 of green plum surface defects reaches 98.95%, of which the 388 classification precision of rain spots is the highest, reaching 389 100.00%. Followed by the classification precision of normal 390 green plums is 99.56%, the classification precision of scars 391 is 98.59%, the classification precision for rot is 98.25%, and 392 the classification precision for cracks is 96.10%.

393
The confusion matrix obtained by the WideResNet50-394 AdamW-Wce network is shown in Figure 4. In the test set, 395 140 of the 142 scar green plum images were correctly clas-396 sified, 1 image was misclassified as rot, and 1 image was 397 misclassified as crack. 393 of the 395 rot green plum images 398 were correctly classified and 2 images were misclassified as 399 cracks. All 226 intact green plum images were correctly clas-400 sified. 74 of the 80 crack green plums images were correctly 401 classified, 1 image was misclassified as scar, and 5 images 402 were misclassified as rot. 389 of the 392 rain spot green plum 403 images were correctly classified, 1 image was misclassified 404  The green plum defect classification network built in this 440 paper was improved on the basis of the WideResNet50-SGD 441 network. Therefore, in order to verify the classification per-442 formance of the model, the green plum data used for the 443 WideResNet50-AdamW-Wce network test was imported into 444 the WideResNet50-SGD network for testing. The momen-445 tum was set to 0.9, the learning rate was set to 1e −4 , 446 and the batch size was set to 64. The model perfor-447 mance of the WideResNet50-AdamW-Wce network and the 448 WideResNet50-SGD network were compared in the loss 449 curve graph and classification performance respectively. The 450 VOLUME 10, 2022  loss curve graph is shown in Figure 6. The test results are 451 shown in Table 5. It could be seen from the loss graph that the network using the 481 AdamW optimizer converged faster. This showed that the use 482 of the AdamW optimizer could achieve a good convergence 483 effect while reducing the loss value faster during the training 484 process.

485
From the test results in Table 5, it can be seen that 486 the WideResNet50-AdamW-Wce network had good perfor-487 mance in each defect classification, and could identify and 488 distinguish the main characteristics of each defect. The four 489 contrasting convolutional neural networks did not all achieve 490 high classification accuracy after training. When the main 491 features were not correctly identified, misjudgment occurred, 492 and the difference between different convolutional neural 493 network structures and network parameters would affect the 494 identification and differentiation of the main features. There-495 fore, the WideResNet50-AdamW-Wce network built in this 496 paper was optimized on the basis of the WideResNet50 net-497 work. It can be seen from Table 5 that the accuracy of the 498 WideResNet50-AdamW-Wce network for the classification 499 of green plum surface defects reaches 98.95%. Compared 500 with the WideResNet50 network, the classification accuracy 501 of the former was increased by 2.77%. The WideResNet50-502 AdamW-Wce network had a certain improvement in the 503 defect classification accuracy of rot, rain spot, scar, crack 504 and normal green plums. Compared with the WideResNet50-505 SGD network, the precision rates were improved by 4.75%, 506 2.54%, 1.33%, 11.04% and 1.58% respectively. Among 507 them, the classification precision of cracks has the highest 508     Table 6, Table 7, Table 8 respectively.

526
It can be seen from, Table 6, Table 7, and Table 8 that  The proposed Resnet network provides a new direction 541 for solving the problems of gradient disappearance, gradient 542 explosion and degradation caused by the deepening of convo-543 lution layer and pooling layer. Comprehensive analysis, based 544 on the WideResNet50 model, the WideResNet50-AdamW-545 Wce model proposed in this study has the advantage of 546 using a 3 × 3 small convolution kernel, which can iden-547 tify richer features and increase feature discrimination. The 548 parameters were normalized by BN to reduce the occur-549 rence of excessive changes in parameters due to the different 550 amount of surface defect data of different plums. In order 551 to reduce the false detection problem caused by the high 552 similarity between rot and other defects, the WceLoss loss 553 function can be used to improve the discrimination of fea-554 tures, thereby improving the classification accuracy. The data 555 set composed of 12350 green plum surface defect images 556 obtained after data enhancement has a large amount of data. 557 Using AdamW optimizer, it can achieve good convergence 558 effect quickly and smoothly. Compared with ResNet50-559 SGD, WideResNet50-SGD, WideResNet50-SGD-Wce and 560 WideResNet50-AdamW network models, WideResNet50-561 AdamW-Wce model sacrifices some test time, but the 562 recognition accuracy of the four defects of plum has been 563 improved, which can fully meet the detection efficiency 564 requirements of enterprises. WideResNet50-AdamW-Wce 565 network learns more about greengage defect features 566 VOLUME 10, 2022

IV. CONCLUSION
surface defect, only 103ms, it is enough to meet the 1800 per 623 hour requirement.

624
Based on the static classification of green plums sur-625 face defects, this paper completed the construction of the 626 WideResNet-based green plum surface defect classification 627 model WideResNet50-AdamW-Wce, and obtained a better 628 surface defect classification result. Under dynamic condi-629 tions, it can meet the requirements of enterprises to detect 630 1800 pieces per hour. In the subsequent research, multi-view 631 vision technology should be considered to obtain the full 632 surface image of green plum. The three-dimensional model of 633 green plum can be obtained by three-dimensional modeling, 634 which can reduce the influence of curvature change caused 635 by three-dimensional entity dimensionality reduction into 636 two-dimensional image on surface defect characteristics.