SPECIAL SECTION ON DATA MINING FOR INTERNET OF THINGS

Crop disease diagnosis is an essential step in crop disease treatment and is a hot issue in agricultural research. However, in agricultural production, identifying only coarse-grained diseases of crops is insufficient because treatment methods are different in different grades of even the same disease. Inappropriate treatments are not only ineffective in treating diseases but also affect crop yield and food safety. We combine IoT technology with deep learning to build an IoT system for crop fine-grained disease identification. This system can automatically detect crop diseases and send diagnostic results to farmers. We propose a multidimensional feature compensation residual neural network (MDFC–ResNet) model for fine-grained disease identification in the system. MDFC–ResNet identifies from three dimensions, namely, species, coarse-grained disease, and fine-grained disease and sets up a compensation layer that uses a compensation algorithm to fuse multidimensional recognition results. Experiments show that the MDFC–ResNet neural network has better recognition effect and is more instructive in actual agricultural production activities than other popular deep learning models. INDEX TERMS IoT, multiple crops, fine-grained disease recognition, ResNet, singular value decomposition.


I. INTRODUCTION
For a long time, crop disease has been one of the most urgent problems in the field of agriculture.It directly affects crop yield, food safety, and sustainable development.South American rubber blight prevents exploitation of native rubber trees, and wheat stem rust strain Ug99 spread across Africa, Asia, and the Middle East.Historically, the worst pest is the Irish potato blight, which killed 1.2 million people from 1845-1849 [1].Oerke [2] calculated total global potential loss from 2001-2003 at 26%-29% for soybean, wheat, and cotton; 31% for maize; 37% for rice; 40% for potatoes.
The associate editor coordinating the review of this manuscript and approving it for publication was Chun-Wei Tsai .
Rapid and accurate identification of crop diseases is the first step prevention and control.Early identification limits the damage and allows for less intensive countermeasures.If crop diseases are inaccurately recognized, then treatments may be ineffective or even harmful to crops.Globally, and especially in developing nations, methods for identifying crop diseases are mostly manual.Farmers recognize diseases on the basis of tradition and limited training with potentially high error rates.Similarly, they may not have access to latest information about crop disease treatments.Even when visual inspection is performed by experts in accordance with detailed guidelines and standards, significant interrater variability and low intrarater repeatability are still found [9]- [14].Therefore, misdiagnosis of crop diseases and inappropriate treatment methods are common and can severely affect agricultural production.Inaccurate chemical treatments can be ineffective, and unnecessarily high dosages increase cost and cause pollution.We must use scientific control technology.
Some scientific detection methods are direct.Samples of plants are analyzed in laboratories with techniques, such as polymerase chain reaction, immune fluorescence, fluorescence in-situ hybridization, enzyme-linked immunosorbent assay, flow cytometry, and gas chromatography-mass spectrometry [15].Indirect methods include thermography, fluorescence imaging, and hyperspectral techniques [16].
With the development of artificial intelligence technology, deep learning has been used in many fields.In agriculture, deep learning technologies have been widely used in obstacle detection, fruit counting, crop yield estimation, field soil moisture prediction, weather prediction, crop disease identification, and other production activities [17]- [25].Specifically, crop disease recognition has been a frequent subject of research in recent years [16], [21], [26]- [28].In crop disease recognition, diseases in fruits, vegetables, and cash crops are mainly used for identification [17].
Analysis of current methods for identifying crop diseases based on deep learning found that the existing methods still have certain limitations.The success of these models is dependent on the quality of the data set [17], [29], color spectrum and vegetation indices [28], different stages of the disease [30], and the limited availability of data sets [29].The neural networks often have the following limitations: 1.) Existing methods often use a single type of crop (such as tomatoes [21] and cucumbers [31]), and few methods support multiple crops and diseases.
2.) Identification process focuses on recognizing diseases without clear indication of the severity of the disease.In agricultural production, the degree of crop disease is also essential.It directly determines the type and level of treatment.Inaccurate type or dosage not only affects the efficacy but also the safety of consumers [32] and causes unnecessary environmental damage [33], [34].
3.) Existing methods mostly focus on shallow neural network models, such as AlexNet and VGG.Although they have achieved good recognition accuracy for diseases, they need more sophisticated recognition for the disease level.
To solve these limitations and contribute to agricultural production, we have combined deep learning with IoT technology to build an agrarian IoT system for crop disease identification.In the deep learning module of the IoT system, we constructed the multidimensional feature compensation residual neural network (MDFC-ResNet) with feature compensation.Compared with the existing methods of crop disease identification, our model can identify the severity of crop diseases; it is instructive in actual agricultural production activities.Our contributions are as follows: 1.) On the basis of deep learning and IoT technology, we build an end-to-end IoT system for crop disease identification.This system can obtain crop disease information in time and feed it back to farmers.
2.) We improve the model training by using singular value decomposition (SVD) technology to prepare the data.We use SVD to process images of crop leaves, extract relevant information, eliminate noise, compress data size, and reduce image size to a certain extent.
3.) We optimize the network model by adjusting the initialization and optimization procedures of the residual network.
The remainder of the paper is arranged as follows: Chapter II describes the related literature; Chapter III introduces our IoT system and the residual network of the multidimensional feature compensation mechanism proposed in this article; Chapter IV analyzes the experimental results, and Chapter V describes the conclusions and recommendation for future research.

II. RELATED WORK
Early crop disease identification methods are mostly manual, mainly by farmers or related experts to diagnose and identify crop diseases in the field [13], [35].These methods are very dependent on the farmers or relevant experts' own experience in identifying crop diseases.The problems of these methods are as follows: keen personal subjective awareness, low recognition efficiency, and high recognition error rate.
With the development of image processing technology and its application in the agricultural field, crop disease identification has improved [18], [19], [36].Computer image processing refers to converting an image signal into a digital signal and then processing it by using a computer.Its advantages are good reproducibility, high processing accuracy, rich processing content, complex nonlinear processing, and flexibility.However, in solving complex problems of crop disease identification, image processing technology seems to be inadequate, and the recognition accuracy rate cannot reach the expected effect.
To solve the problem of crop disease identification in agricultural production activities, a large number of scholars introduced deep learning [20]- [25] into the agricultural field and achieved excellent results.In deep learning technology, photos of crop disease parts are captured, and then a neural network model is sent via the computer.The photos of crop disease parts are sent to the neural network model for learning (feature extraction), and the learned model is finally used to identify crop diseases.Deep learning technology is faster, more convenient, and has higher recognition accuracy than the first two methods.
Recent works use the combination of IoT technology and deep learning technology to identify crop diseases [37]- [39].This method uses IoT technology to combine various information sensing devices with the Internet to collect information in real-time and feed back to the deep learning model.The deep learning model processes the collected information in time and then displays the results in smart terminal devices.In this way, we can grasp relevant information in real-time.An artificial intelligence brain is also available to analyze the information.We combine the IoT technology with deep learning to build an IoT system for crop disease identification in real-time.In the next section, we will focus on the structure of the experimental system.

III. PROPOSED SYSTEM AND MDFC-ResNet SCHEME A. SYSTEM STRUCTURE
The proposed IoT system combines video cameras, deep learning models, and intelligent terminal devices.The system uses cameras to collect crop videos.It feeds back the health status of crops analyzed by deep learning models to farmers through smart terminal applications (web applications or smartphone applications).Fig. 1 shows the system structure.The system consists of the following six parts: one or more video cameras, decoder, deep learning model, message center, processing center, and terminal (computer or smartphone).The main functions of each par and the system operation process are as follows: The video camera is set up in crop-fields or greenhouse to collect crop information.Usually, we set up multiple video cameras.
The decoder can receive data from multiple video cameras, decode the video data, and extract the crop image from the decoded information stream.
The deep learning model receives the crop image from the decoder, judges the health status of the crop through the trained model, and sends the result to the message center.
The message center receives the discrimination results and organizes and manages the discrimination results in the form of the message queue, which the processor uses.
The processor obtains information from the message center, processes the information, and sends it to the web application and smartphone application in the form of notification.
Deep learning models are an essential part of the system, directly determining the performance of the IoT system.At present, most crop disease identification systems only identify the types of crop diseases but do not perform more fine-grained identification of these diseases (the degree of crop disease identification).However, in actual agricultural production activities, the degrees of crop disease are different.Thus, the treatment plan adopted and the amount of medicine used are also different.Fine-grained identification of crop diseases is instructive in terms of disease treatment, reducing the number of pesticides used, and protecting the crop and the natural environment.Therefore, we propose the MDFC-ResNet model, which can identify the general and serious diseases of crops and is more instructive in actual agricultural production activities, for the system's deep learning model.The MDFC-ResNet model is described in detail in the following section.

C. DATA PROCESSING
The data set used in this study is obtained from AI Challenger [40].It was divided according to ''species-diseasedegree,'' with a total of 59 categories, including 10 species, 49 detailed disease categories, and 10 health categories with a total of 36258 pictures.Each image was obtained from a crop in a natural environment and treated with only one leaf.Fig. 3 shows the various class images.
The dataset was not ready for use in image classification due to different sources of pictures in the original dataset, the shooting environment, equipment, and differences between the crop species.Substantial differences in the number of picture types, uneven picture quality, and inconsistent picture sizes cause image recognition problems.To solve these problems, we processed the dataset before model training.The process included three critical steps, namely, data enhancement, data normalization, and SVD.
The first step of data enhancement solves the problem of significant differences in the number of pictures between the   original data set.Random cropping might remove the diseased part of the picture, resulting in loss of the characteristic information.Therefore, we used rotation and horizontal flip to enhance the dataset.We expanded categories with less than 1,000 pictures to approximately 1,000 and reduced larger categories to approximately 1,000.Balancing the number of images in various categories removes the impact of the number of pictures on the final classification accuracy.Fig. 4 shows an example of data enhancement process.The original image is flipped horizontally and rotated by 90 • , 180 • , and 270 • .At the end of this phase, the dataset increased from 36258 to 63265 images.
The second step of data normalization converts all images to a uniform size to facilitate model training.We normalized the pictures in the dataset to 224 × 224 pixels prior to the experiment.Many deep learning models use this image size.
The third step of singular value decomposition solves the problem of picture quality.It extracts important information from the original picture and removes noise.In many images, a small section of data carries most of the information, and the remainder is irrelevant.The quality of the pictures in the original data is uneven.We adjust the singular value and observe the effect on the image, an example of which is shown in Fig. 5.We select the singular value 0.9 to process the images in the data set.An appropriate initializer sets initial weights that allow the network to train in an efficient and timely manner.
The goal of deep learning is to continuously adjust network parameters to allow them to perform various nonlinear transformations on the input to fit the output effectively.Essentially, it is the process of solving the optimal solution of the loss function.Research on deep learning focus on algorithms to update the parameters.We refer to these algorithms as optimizers.Selecting an optimizer in the field of deep learning is one of the top priorities of a model.Even when the dataset and the model architecture are the same, using different optimizers may lead to different training results.Therefore, the combination of network and dataset is tested using different optimizers to select the most effective approach.
With the development of deep learning in recent years, some new optimizers and initializers have been proposed and applied [41].Therefore, before the experiment, the model is tuned to select the appropriate initializer and optimizer.This experiment is performed in a GPU environment, using Keras framework based on TensorFlow, which mainly adjusts the model's learning rate, epoch, and batch parameters.Table 1 shows the experimental environment and parameter configuration.We use 58725 pictures in the dataset during the experiment.We use training samples and 4,540 pictures as test samples.
The training samples are divided into the training set and the verification set with a ratio of 8:2, as shown in Table 2.

1) INITIALIZER SELECTION
This experiment selects the optimal initializer for the traditional residual neural network.In the experiment, three common initializers, namely, Lecun et al. [42], Glorot and Bengio [43], and He et al. [44], [45] are selected as the research objects.Each initializer is used with normal and uniform distribution, resulting in six types, namely, lecun_normal, lecun_uniform, glorot_normal, glorrot_uniform, he_normal, and he_uniform.Technical specifications can be found in the Keras initializer documentation [46].On the basis of a learning rate of 0.0001, the optimizer selects six sets of comparative experiments.Table 3 shows the experimental results.The glorot_normal initializer proved most suitable for the residual network given the highest scores for training accuracy, validation accuracy, and test accuracy.Therefore, we selected the glorot_normal initializer as the model's initializer for the remaining experiments.

2) OPTIMIZER SELECTION
This experiment aims to select the best optimization algorithm for the traditional residual neural network.We compared five optimizers by using the initializer, glorot_normalas.The five optimizers are SGD [47], RMSProp [48], Adadelta [49], Adam [50], and Adamax [51].Technical specifications can be found in the Keras optimizer documentation [48].The results of the comparative tests are shown in Table 4.The experimental results show that the Adam optimizer scores best in training accuracy and test accuracy and second best in validation accuracy.Therefore, we select Adam optimizer for this experiment.

E. MDFC-ResNet
Our MDFC-ResNet is based on the deep residual network optimized in the previous section.The MDFC-ResNet model consists of three dimensions, namely, species, disease, and disease level.A compensation layer connects the three levels.The recognition results of species and disease dimensions are fed back as a compensation and error correction mechanism and to improve the accuracy of crop disease level recognition.Fig. 6 shows the structure of our deep residual neural network with the multidimensional compensation mechanism.The uppermost layer shows the first species dimension.The optimized ResNet-34 network is used to identify the species to which the picture belongs.The second dimension is the disease dimension in the middle layer of Fig. Algorithm 1 describes the process of MDFC-ResNet.For any single crop disease picture, the output of the three dimensions is the probability distribution matrix of the crop species to which the picture belongs, the disease, and the level of disease.A compensation layer is used after three dimensions.The compensation layer receives the probability distribution matrices from the three dimensions.It uses the probability distribution matrix of the identified species and t of the disease as feedback data to compensate for the probability of the obtained level of disease.
Algorithm 1 introduces the specific design scheme of MDFC-ResNet.Fig. 7 shows the operation flow of the distribution matrix and compensation layer.
The results of the species identification, disease identification, and disease level identification are in different dimensions.Thus, the species dimension and the disease dimension results are ''expanded'' in accordance with the disease level result before all results are fused by using Eq. 1, as follows: In the model, P Xi is the probability of the i-th species, P Yj is the probability of the i-th disease, P Zi is the probability of the i-th disease level, and P Zi is the final detailed feature recognition result.In the process of selecting α and β values, we set α = 1, determine the value of β, and finally, determine the best value of α.After several trials, we achieve the highest accuracy of the test set when α = 10 and β = 1.5.
Algorithm 2 shows the calculation algorithm of the compensation layer.
The specific process of ''expanding'' is shown in Fig. 8, with apples and tomatoes as examples.The recognition result of the species dimension is a matrix of 2×1, which represents the probability of apple and tomato.The recognition result of the disease dimension is a 4 × 1 matrix.The matrix contains apple scab, apple gray spot, tomato powdery mildew, and tomato scab.The disease level dimension result is an 8 × 1 matrix, which generally contains apple scab, severe apple scab, general apple gray spot, severe apple gray spot, general tomato powdery mildew, severe tomato powdery mildew, general tomato scab, and severe tomato scab.At this point, the dimensions of the matrices are different.In the expansion process, the size of the disease level dimension is used as a reference, and the results of the species dimension and the disease dimension are expanded into 8 × 1 matrix.The species dimension extends from 2 × 1 to 8 × 1; thus, each element repeats four times before moving to the next element.Similarly, because the disease dimension expands from 4 × 1 to 8 × 1, each element of the disease dimension repeats twice before moving to the next element.At the end of this expansion process, the three matrices have identical sizes.

IV. PERFORMANCE ANALYSIS
We preprocess the data set in the model tuning process by normalizing the size of the pictures.We use SVD technology to remove the noise in the picture and determine the model's initializer and optimizer.This section focuses on the performance of our proposed model.
In the experiment, we compare MDFC-ResNet with commonly used methods in crop disease identification, namely, AlexNet, VGG, and ResNet-50.The results of the experiment confirm that our proposed residual network with multidimensional feature comparison performs better than traditional models.The overall results are shown in Table 5.The table shows that our MDFC-ResNet has the best performance in terms of training accuracy, validation accuracy, and test accuracy.Specifically, the accuracy of the training set is increased by 1.10%, 8.40%, and 5.31%; the accuracy of the validation set is increased by 0.08%, 4.12%, and 3.04%; the accuracy of the test set is improved by 5.03%, 2.12%, and 3.18%.Our model contains two ResNet-50 models and one ResNet-34 model (arguably less accurate than ResNet-50); thus, we conclude that the compensation layer adds accuracy to our model is reasonable.
Network performance is not only measured by high accuracy.Accuracy is simply the ratio of accurately predicted values to the total number of observations.In any recognition model, we have true positive and true negative values, as well as false positive and false negative values.Precision is the ratio of true positive over true positive and true negative.

Algorithm Compensation Layer
Input: Species result matrix, disease result and disease level result matrix Output: Picture recognition results 1: For disease level result i in disease level result matrix 2: Find the species s to which i belongs to 3: From species result matrix, find species s result 4: Set position i of the extended species result matrix as species s result 5: Find the disease c to which i belongs 6: disease result matrix, find disease level result 7: Set position i of the extended disease matrix as disease c result End For 9: Combine the extended species result matrix, the result matrix, and the disease level result matrix by using Formula 1 calculate the detailed result matrix of compensation 10: Find the maximum score from the detailed result matrix of compensation 11: Output The category corresponding to the maximum score 12: End Algorithm 2 In our study, it represents the accurately identified diseased leaves out of the total diseased leaves.Recall or sensitivity is the ratio of true positive, overall labeled as positive, whether correct or not.In our study, it represents the accurately identified diseased leaves out of all leaves labeled as diseased.Finally, F1 ratio is the weighted average of precision and recall.This measure is more useful than accuracy if the class distribution is uneven.Our model holds 59 crop disease categories, but 49 of which are levels of specific diseases; only 10 categories show healthy leaves.This finding skew the disease identification, but the F1 statistic corrects the disparity.For a more detailed analysis, we review the precision, recall, and F1 values of all four models in the 59 classifications.Table 6 shows the precision, recall rate, and F1 measure of each model.
We compare the performance of MDFC-ResNet with the best performance of the other three models, in terms of ranges of precision, recall, and F1.For each network and each statistic, we record the average, minimum, and maximum scores.We then determine the highest average for the three other networks and compare with the average of MDFC-ResNet.The same procedure is performed for the minimum and maximum scores.All networks had perfect 100% scores; thus, we considered them as well.Table 7 shows the results.MDFC-ResNet performs better than the other models on all measures.It has the highest average accuracy, the highest range (minimum to maximum), and perfect scores for precision, recall, and F1 values.The only exception is the minimum score for F1 percentage, and class 22 obtain the worst case; the score of the two other models is only 44.45%.Finally, MDFC-ResNet scores lower than 50% for F1 in only two of 59 cases, namely, 22 and 50.

V. CONCLUSION AND FUTURE RESEARCH
We have shown that our IoT system is effective in crop disease recognition systems of the agricultural industry.Through the combination of deep learning and IoT technology, the proposed method can be automatically used with multiple crop types.It differentiates between levels of disease, apart from recognizing the disease.Treatment protocols may differ between mild and severe cases of crop disease, especially in types and amounts of chemicals used to combat the disease.Most importantly, accuracy is high, even compared with human recognition.We attribute much of this accuracy to using three stages of recognition and the compensation layer.
Future research can focus on two aspects; on the one hand, identifying reasons why recognition fails in some cases.Improving accuracy at the low end enhances the efficiency of the system.Picture quality may be a factor in some cases, and we can establish guidelines and requirements for pictures in the dataset.One possible solution may be the use of color calibration charts to be included in the images.Preprocessing may then include correcting the image colors to standard.Visual attention mechanisms and target detection can then improve picture data.On the other hand, we improve the IoT system by increasing sensor types.Through sensors, we can collect weather, soil, and air quality data to improve the accuracy of crop disease recognition.
ResNet is suitable for multiple purposes in agriculture.Farmers can use them for fruit counting, crop yield estimation, field soil moisture prediction, weather prediction, crop disease identification, and other production activities.Surprisingly, little is known about the actual level of use.We may learn from the failure of medical diagnostic systems in healthcare.One reason is the difficulty of data entry.This case is not the same as in agriculture.Farmers can take pictures with their cell phones and send them for analysis.Another factor may be more relevant.Medical diagnostic systems suffer from lack of follow-up in the form of recommendations for treatment.Future research could focus on the impact of providing treatment recommendations according to best practices.Combined with the excellent performance of IoT technology and neural convolutional networks, these steps may improve penetration in practice.

Fig. 2
Fig. 2 shows the overall process of our experiment.It consists of two parts.In the data processing phase, we perform data enhancement, normalization, and SVD operations on pictures in the data set.The purpose is to reduce the negative impact of the data set on model training.In the model training phase, we divide the data set into a training set, a validation set, and a test set.The training set trains the model.The validation set verifies that the expectations are satisfied; if yes, then the model is saved, and if no, then the parameters in the model training are adjusted until the expectations are satisfied.The test set is used to test the accuracy of the model.

6 .:
It uses the optimized ResNet-50 network to identify diseases.The third dimension is the disease level dimension in the bottom layer of Fig. 6.The ResNet-50 models of disease dimension and disease level dimension use parameter sharing to speed up the recognition process and improve recognition accuracy.Algorithm 1 MDFC-ResNet Input: Training dataset and validation dataset Output: Picture recognition results 1For image in train dataset 2: Picture feed into the first (species) dimension 3: ResNet-34 4: Result of species identification 5: Picture feed into the second (disease) dimension 6: ResNet-50 7: Obtain disease results 8: Picture feed into the third (disease level) dimension 9: ResNet-50 10: Get disease level results 11: Use compensation layer to integrate species results, disease results, and disease level results 12: End For 13: Output Final results 14: End Algorithm 1

FIGURE 7 .
FIGURE 7. Operation process of compensation layer.

FIGURE 8 .
FIGURE 8. Apple and tomato as examples to illustrate the operation process of the compensation layer in detail.

TABLE 1 .
Experimental environment and parameter configuration.

TABLE 2 .
Data set partition.

TABLE 3 .
Experimental results of each initializer.

TABLE 4 .
Learning rates of five optimizers and experimental results.

TABLE 5 .
Experimental results of each model.

TABLE 6 .
Results for each class in MDFC-ResNet.

TABLE 7 .
Summary of accuracy.