Cybersecurity Status Assessment of Cloud Manufacturing Systems Based on Semiquantitative Information

The network security status assessment (NSSA) method can evaluate the network security status of cloud manufacturing systems (CMSs), which is of great significance to reduce the network security risk and loss of CMSs. At present, the NSSA of CMSs suitable for semiquantitative and uncertain information conditions is commonly used, which has certain limitations and low accuracy. This paper proposes an NSSA method for CMSs based on semiquantitative information. First, through the detailed analysis of the influencing factors of the network security status of the CMSs, the security evaluation indicators are selected, and the evaluation framework containing multilevel indicators is established by combining semiquantitative information. Second, the evaluation level is established according to the data distribution and the properties of the indicators. Third, a process of data fusion reasoning based on an evidential reasoning algorithm is designed, and a strong reference fusion case is given. Finally, the effectiveness of the proposed security state assessment algorithm in the NSSA of CMSs is verified and analyzed by simulation experiments. The experimental results show that the proposed method can make full use of semiquantitative information and uncertain information to evaluate the network security status of CMSs, and the evaluation results can reflect the actual security status of CMSs.

shortcomings due to excessive reliance on expert knowledge and human subjectivity. For example, the lack of experience of expert knowledge may directly lead to the serious deviation of evaluation results from the true value, and when incomplete information and semiquantitative information are involved, accurate evaluation results cannot be obtained, and even the computational complexity is increased in vain [14].
Data-driven evaluation methods include the Bayesian inference analysis method and machine learning-based evaluation method. The former uses prior knowledge to obtain conclusions by statistical sample information, which can address uncertain problems [15]. Then, knowledge reasoning is used to solve the problem, which can accurately identify network attacks and comprehensively and flexibly evaluate network security conditions [16]. However, the evaluation method based on data-driven methods cannot deal with semiquantitative information, the demand for evaluation data is large, the quantitative information evaluation results of small-scale samples are not accurate, and the accurate results need to be repeatedly trained, which not only increases the difficulty of training but also has the problem that the optimization principle of model training cannot be explained. Although expert knowledge can compensate for the lack of model training, it will not improve the evaluation accuracy, so it cannot reflect the real advantages of expert knowledge [17].
The existing network security evaluation methods mainly have the following shortcomings: (1) Some methods cannot process semiquantitative information, or the computational complexity is too high when processing semiquantitative information.
(2) Some methods strongly rely on expert knowledge or human subjectivity, which cannot effectively use expert knowledge and even seriously affect the accuracy of the evaluation results.
(3) Some methods cannot accurately deal with uncertain information in the evaluation process.
In summary, this paper proposes a network security status assessment (NSSA) method for CMS based on semiquantitative information, which can comprehensively use qualitative knowledge and quantitative information to fuse different types of data and can also express the uncertainty information of various security assessment indicators in CMS. According to the state data of the CMS, a new security state evaluation framework is established, and the network security level of the CMS is evaluated by referring to the national network security evaluation level standard.

II. PROBLEM DESCRIPTION
The cyber risk of CMS is the degree of impact on the interests of individuals, organizations or countries due to potential threats or with a certain probability. The judgment of the degree is often determined according to the loss characteristics. Assessing the cyber risk of CMS in advance can find and solve problems as early as possible before the VOLUME 11, 2023 production of CMS and maximize the network security of the system. The CMS model involved in this paper is shown in Fig. 1.
In Fig. 1, because the field equipment layer is mostly all kinds of sensors, digital machine tools, mechanical devices, instruments, etc. At the same time, the incompatibility 43460 VOLUME 11, 2023 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply.
between various industrial protocols further aggravates the heterogeneity of data.
In Fig. 1, the field network layer is based on industrial Ethernet and uses various wireless communication technologies to realize the transmission of industrial data and instructions. Field management mainly realizes the management function of equipment, production, data, documents and assets within the manufacturing enterprise. The application service layer is mainly oriented to cloud services and uses the cloud platform to connect multiple CMSs. The user layer is mainly for individuals, organizations, governments, ecological enterprises and other users to access the CMS to complete mutual coordination and customization services.
The heterogeneity of the whole CMS structure increases the complexity of the system. When the industrial system that used to be an ''isolated island'' is connected to the cloud platform, it is bound to cause people's concern about its security. Since different levels in Fig. 1 involve different technologies and the security issues among the technologies are complex, this paper analyzes and evaluates the security status of CMS from a data perspective.
Due to the large differences in equipment, protocols, working principles and technologies of each layer, the dynamic data of each layer also have different attributes and characteristics, and the data itself can reflect the network security status of this layer to varying degrees. Therefore, the network security status of the CMS can be evaluated by analyzing the data of each layer of the CMS and scientifically selecting the core attribute data and indicators.
There is a large amount of semiquantitative information in the CMS, and it can reflect the correlation, constraints, fuzziness and uncertainty among the data of the network security status, which will increase the difficulty of assessment [18]. Therefore, the evaluation method itself should have the ability to comprehensively use semiquantitative information and deal with various uncertain information [19].

III. RELATED TECHNOLOGIES A. SEMI-QUANTITATIVE INFORMATION
Semiquantitative information refers to the information that contains quantitative data and qualitative knowledge. Quantitative data refer to the specific data that can be expressed as a certain amount or range, and they can be directly obtained by monitoring equipment. Qualitative knowledge is subjective and abstract information that cannot be measured or collected and can only be acquired by expert knowledge through the evaluation and perception of complex systems [20].

B. ER ALGORITHM
The evidential reasoning (ER) algorithm is a method to fuse multisource information based on decision theory and the Dempster combination rule in D-S evidence theory, which is suitable for dealing with semiquantitative information and various uncertain information [21]. Compared with D-S evidence theory, the linear calculation process of the ER algorithm can not only reduce the computational complexity but also solve the conflict problem between evidence attributes [22]. At present, the ER algorithm has been applied in typical applications. For example, Zhou et al. applied it to oil pipeline leakage fault monitoring [23], Qiu et al. applied it to relay fault diagnosis [24], Bi et al. applied it to weapon equipment system effectiveness evaluation [25], and it was applied to relay fault diagnosis. Wang et al. applied it to the safety assessment of natural gas storage tanks [26]. In addition, the ER algorithm has significant advantages when evaluating the network security status of CMS fusing a large number of different types of information. The detailed steps of the ER algorithm are described in [27] as follows: It is assumed that there are basic attributes {γ 1 , γ 2 , . . . , γ i , . . . , γ M } constituting a multilevel evaluation system, where {ω 1 , ω 2 , . . . , ω i , . . . , ω M } represents the weight of the basic attribute, and 0 ≤ ω i ≤ 1. The output is rated N . Then, the basic steps of the ER algorithm are as follows: (1) After the value of the evaluation attribute is determined, it is necessary to calculate the corresponding confidence degree to obtain the basic probability quality. As shown in Formula (1).
where U (r i ) represents the value of attribute r i , and R i,j represents the reference value of attribute r i at the j evaluation level.
where P i,j represents the basic probability quality relative to the evaluation level j, P i,θ represents the basic probability setting of the evaluation set of i that is, the residual probability attribute without the i basic attribute of the assigned result, P i,θ = Q i,θ + P i,θ . Q i,θ represents the unassigned basic probability mass with respect to the incompleteness of the i-th fundamental attribute, and P i,θ represents the unassigned basic probability mass with respect to the insignificance of the i-th fundamental attribute.
(3) The probability quality of the j valuation level can be obtained by combining the first i basic attributes with VOLUME 11, 2023 43461 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply. evidence theory. The steps are shown in formulas (6) ∼ (9).
where P I (i),j represents the probability quality of the j evaluation level after the combination of the first i basic attributes. It can be obtained by the following formula (10).
(4) Based on the obtained probability quality, the confidence degree σ j of the j-th evaluation level and the residual confidence degree σ θ of the unset evaluation result are calculated, as shown in formula (11) and formula (12).
The above are the basic steps of the ER algorithm.

IV. NSSA FRAMEWORK OF CMS
To solve the problem of the NSSA of CMSs, this paper proposes an NSSA method for CMSs based on semiquantitative information. The system network security status data are analyzed quantitatively and qualitatively, and then the ER algorithm is used to fuse the data step by step, compare the results with the predefined level division, and finally determine the assessment result of the CMS.
To facilitate the description of the framework evaluation process of this paper, the field management layer and the field device layer are selected from Fig. 1 for evaluation, and each layer constructs an evaluation indicator system from three aspects: equipment security, service quality and network security.
(1) Select evaluation indicators. In the process of selecting evaluation indicators, it is necessary to ensure that the selected indicators can objectively reflect the real network situation of the CMS and meet the relevant requirements of system security to ensure the availability of the selected indicators. Evaluation indicators can be divided into two categories: one category of indicators is quantitative data, and the other category of indicators is qualitative knowledge. To improve the accuracy of the evaluation, first, according to the specific objects involved in each layer of the CMS, the important and typical core indicators are scientifically analyzed and selected to construct the evaluation framework. The qualitative indicators are then quantified to make them available for quantitative analysis.
To facilitate the description of the framework evaluation process of this paper, the field management layer and the field device layer are selected from Fig. 1 for evaluation, and each layer constructs an evaluation indicator system from three aspects: equipment security, service quality and network security.
Assume that Y is the overall indicator set reflecting the network security status of the CMS, which is defined as Y = [y 1 , . . . , y M ] T . The network security status of CMS is evaluated from three dimensions: network equipment security, service quality and network security. Assume that set y is the security evaluation indicator of network equipment, which is defined as y = [y 1 , . . . , y L ] T , y ⊆ Y ; Set e is the service quality evaluation indicator, which is defined as e = [y 1 , . . . , y N ] T , e ⊆ Y ; Set g is the network security evaluation indicator, which is defined as g = [y 1 , . . . , 2) Establish an evaluation framework. The NSSA indicators in CMS are related to each other. When improving the accuracy of assessment results, we cannot just rely on the method of processing a single piece of information but comprehensively consider and reasonably use semiquantitative information to build a more accurate assessment framework.
3) Establish a model to evaluate the network security status of the CMS. Its model construction is shown in Equation (13). Table 1 shows the meanings of specific parameters in Formula (13).

A. EVALUATION METRICS
The evaluation indicators are divided into quantitative indicators and qualitative indicators, and each indicator takes into account all kinds of current standards as much as possible according to the specific circumstances. CMS has the characteristics of high complexity and low security due to its network heterogeneity, component diversity, high confidentiality and high added value of commercial big data. Its centralized storage makes CMS more likely to become a high-risk area of network attacks.
By selecting reasonable security indicators that meet the evaluation requirements to evaluate the network security status of the CMS, it can understand its own security level and risk level and warn security managers in time.
In this paper, the network security status evaluation indicators of CMS are selected from the three dimensions of equipment security, service quality and network security.  The evaluation indicators involved in equipment safety are the mean time to failure, mean running time, failure repair time, failure frequency and failure severity of equipment. The core of a CMS is service, which aims to provide reliable manufacturing services for users by relying on stable network resources and system components. Therefore, the sudden drop in service quality can reflect the security of the system, so the service quality can also reflect the network security status of the CMS, and the evaluation indicator of service quality is as follows: service integrity, average service response time, service access success rate, and average service time.
According to the security threats faced by the network, the security evaluation indicators are divided into network attack frequency and network attack severity. Fig. 2 shows the architecture of the evaluation metrics proposed in this paper.

B. NETWORK SECURITY STATUS EVALUATION FRAMEWORK OF CMS 1) EVALUATION FRAMEWORK A
According to the evaluation indicator architecture in Fig. 2, this section combines the actual working conditions of the CMS and the reasonable analysis of the importance of cloud security and establishes a three-level evaluation framework A of CMS network security status with the selected important security indicators as the evaluation objects. Considering the threat of cyber attacks to CMS, the addition of the 3 rd level indicators to the framework later ensures the rationality of the evaluation framework. Each layer of the framework contains both quantitative data and qualitative knowledge, that is, semiquantitative information. The ER algorithm can fuse semiquantitative information to obtain the NSSA results of the CMS.
In the network security evaluation part, the network attack types of the Edge-IIoTset dataset [28] are divided into Backdoor attack frequency, DDoS attack frequency, Password attack frequency and so on according to the network attack frequency. According to the severity of network attacks, it is divided into Backdoor attack severity, DDoS attack severity, Password attack severity and so on. At the same time, each evaluation indicator is marked with ''r'', and then the weights of different indicators are calculated according to the entropy weight method, denoted as ''ω'', to construct the network security status evaluation framework of the CMS, as shown in Table 2.

2) EVALUATION FRAMEWORK B
This section establishes another three-level evaluation framework B for the network security status of the CMS. Different from Framework A, the network security evaluation part takes the attack types in the network part of the TON-IoT dataset [29] as samples and selects representative network attacks to participate in the evaluation of network security. Each attack is evaluated by its attack frequency and attack severity. Evaluation framework B is shown in Table 3.

C. ASSESSMENT LEVELS
This paper refers to the National Internet Emergency Center security indicator classification and uses expert knowledge to divide the fusion results of CMS network security at all levels into five assessment levels, which are as follows: excellent, good, generally dangerous, dangerous and severely dangerous, as shown in Table 4.
This paper also divides each evaluation indicator into five levels, namely, ''excellent, good, generally dangerous, dangerous and severely dangerous''. At the same time, according to expert experience, the evaluation interval of  Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply. quantitative attributes is established to ensure the accuracy of the experimental results.
The evaluation levels of qualitative indicator attributes are also given based on expert experience. The method of using expert knowledge is beneficial to improve the reasoning speed of evidence, reduce the computational complexity, and reflect the real network security status of CMS. The evaluation levels of each security indicator are shown in Table 5.

D. EVALUATION STEPS/ER FUSION DATA PROCESS
According to Table 2 CMS NSSA framework A, the specific assessment steps are given as follows: Step 1: The ER algorithm is used to fuse the data of the 3 rd level indicators, and then the fusion results are classified and fused according to the 2 nd level indicators to obtain the evaluation level of the 2 nd level security indicators.
Step 2: According to the results of the 2 nd level indicators after fusion, the evaluation level of the 1 st level three security indicators is obtained.
Step 3: The indicator data of the three dimensions are finally fused to obtain the network security status evaluation level of the CMS. The detailed reasoning process of indicator data fusion is shown in Fig. 3.
To elaborate the detailed process of the ER algorithm to fuse the indicator data, this section takes the network attack frequency evaluation constructed in Table 2 as an example and uses the Edge-IIoTset dataset as the sample data to give the detailed process of the evaluation by using the ER algorithm to fuse the indicator data.
In the set collection of attack types, the unit of attack frequency is times/hour. Taking the data between 2:00 AM and 3:00 AM, backdoor attacks are 0 per hour, DDoS attacks are 0 per hour, password attacks are 0 per hour, XSS attacks are 333 per hour, Port Scanning attacks are 0 per hour, and Ransomware attacks are 0 per hour. The SQL injection attack is 0 times/hour, and the uploading attack is 87 times/hour.
(2) According to the weights given by equations (2) -(5) and Table 2, the above confidence measures can be converted into basic probability mass, that is P i,j , i ∈ [311, 318], j ∈ [1,5], and the calculation results are shown in Table 7.
Similarly, when i ∈ [311, 318], j = θ, calculate P i,j , P i,j and Q i,j , and the results are shown in Table 8.       Table 9.
(6) The confidence degree of the network attack frequency (r 31 ) obtained by the fusion of the eight-level indicators can be obtained by formulas (11) and (12) as follows: Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply.
Through the above calculation, it can be concluded that the security state evaluation results determined by the network attack frequency (r 31 ) are {excellent (93.35%), good (6.65%), generally dangerous (0%), dangerous (0%), and severely dangerous (0%)}. Indicates an evaluation rating of ''excellent'' for the frequency of attacks between 2:00 AM and 3:00 AM. The correctness of the calculated results is verified by simulation experiments in Section V.

V. SIMULATION EXPERIMENT A. EXPERIMENTAL RESULTS AND ANALYSIS
The calculation process in Section IV-D is the detailed process of using the ER algorithm to fuse the indicator data. The fusion steps of other indicator data are the same as the calculation process in Section IV-D. Due to space limitations, this paper will not elaborate on each one but only gives the evaluation results between 2:00 AM and 3:00 AM. The specific evaluation results of network attack severity (r 32 ), network security (r 3 ), service quality (r 2 ), network equipment security (r 1 ) and network security status (r) of the CMS are shown in Table 10. Table 10 shows that the evaluation grade of network attack severity between 2:00 AM and 3:00 AM is ''E''; network security is assessed as ''E''; the evaluation level of service quality is ''SD''; the evaluation level of network equipment security is ''E''; and the network security status evaluation level of the CMS is ''E''.

1) SIMULATION EXPERIMENT 1
The Edge-IIoTset dataset was selected to complete the evaluation of network attack frequency and network attack severity. The dataset was sorted, screened and counted, and the data from 24 hours were selected for ER fusion. The evaluation results are shown in Fig. 4 and Fig. 5. Fig. 4 and Fig. 5 show that the security state evaluation value between 2:00 AM and 3:00 AM is 0.2. It indicates that there are few network attacks during this period, all devices are running normally, and the system is in a relatively secure state. According to the evaluation level divided in Section IV-C, the security status determined by network attack frequency and network attack severity can be evaluated as ''E'', which is in line with the example calculation results in Section IV-D and the evaluation results of r 32 in Table 10.
The ER algorithm is used to fuse the security situation assessment values of network attack frequency and network attack severity, and the network security assessment results are obtained. In the two parts of service quality evaluation and network equipment security evaluation, the quantitative information was randomly generated according to the actual situation of the CMS, and the qualitative indicators are  specified according to expert knowledge. The results of the network security evaluation, quality of service evaluation and network equipment security evaluation are shown in Figs. 6, 7 and 8, respectively. Fig. 6 shows that between 2:00 AM and 3:00 AM, the security situation assessment value is between 0.2 and 0.3. This indicates that the network attacks on the system during this period are weak, the device is in normal running state, and the system as a whole is in a relatively secure state. According to the assessment level divided in Section IV-C, the state determined by network security can be evaluated as ''E'', which is in line with the assessment results of r 3 in Table 10. Fig. 6 shows that at approximately 19:00 and 21:00 at night, the security situation assessment value is close to 1, indicating that the system is under serious attack and the equipment may not operate normally, which is judged as ''SD''.
Similarly, Fig. 7 and Fig. 8 show that in the same time, the security situation assessment value of service quality is greater than 0.9 and less than 1, and it is evaluated as ''SD''; the security situation assessment value of network equipment  is approximately 0.25, and it is evaluated as ''E''. All results agree with the evaluation results of r 2 and r 1 in Table 10.
The ER algorithm is used to fuse the assessment results of network equipment security, service quality and network security, and the ER algorithm is used to fuse the security situation assessment values of the above three parts again to obtain the final NSSA results of the CMS. The evaluation results are shown in Fig. 9. Fig. 9 shows that the security situation assessment value between 2:00 AM and 3:00 AM is approximately 0.35. According to the evaluation level constructed in Section IV-C, the CMS network status is evaluated as ''E'', which is in line with the assessment result of r in Table 10. At 11:00 AM, the point CMS network security status is the worst, rated as dangerous. That is, the smaller the value is, the closer to the X-axis, and the better the network security status of the CMS.

2) SIMULATION EXPERIMENT 2
The network part of the TON-IoT dataset was selected to complete the evaluation of the network security part. The dataset was sorted, screened and counted, and a total of 96 data points in 4 days were selected for ER fusion. The  data of network equipment security evaluation and service quality evaluation were randomly generated according to the attack frequency and attack severity. The evaluation results are shown in Fig. 10.
Similarly, the final NSSA results of the CMS can be obtained by fusing the security situation assessment values of the three parts. This is shown in Fig. 11.
The analysis of Fig. 11 shows that the security situation assessment value is the largest around Group 43, indicating that the CMS receives very serious network attacks during this period, and the network security status of the CMS can be evaluated as ''D''.

B. COMPARATIVE EXPERIMENT AND ANALYSIS
In this section, common machine learning methods were used to carry out comparative experiments. Seventy-two data points over three days were selected as the training set, and 24 data points over one day were selected as the test set, which was used to communicate with the support vector machine (SVM) and back propagation (BP). The random forest (RF) algorithm was compared with the K-nearest neighbor (KNN) algorithm. The evaluation value obtained by each algorithm is shown in Fig. 12.
By analyzing Fig. 12, the overall change trend of the NSSA value of the CMS of each model is roughly similar. However, the evaluation value obtained by the ER algorithm is more in line with the actual evaluation value and fits the actual situation, because it can make comprehensive use of semi-quantitative information, effectively use the qualitative information such as expert knowledge, and deal with the uncertain information in the evaluation process. However, machine learning algorithms can only evaluate the network security state of cloud manufacturing systems based on quantitative information, and cannot effectively use qualitative knowledge. The evaluation values and results of the network security status of the CMS for each algorithm are shown in Table 11. Table 11 shows that the RF and KNN algorithms have large evaluation values from 15:00 to 16:00, which is ''D''; SVM and BP have small evaluation values from 5:00 to 7:00, which is ''G''; SVM, BP and KNN have multiple low evaluation values from 19:00 to 23:00, etc. These results have large errors with the actual results. This may not be more accurate to obtain the network security status of the CMS.
In conclusion, compared with the RF, BP, SVM and KNN models, the industrial control heterogeneous network security assessment method based on semiquantitative information has higher assessment accuracy and can more accurately evaluate the network security status of CMS.

VI. CONCLUSION
At present, in the field of network security assessment, there is a lack of methods to reasonably assess the network security status of complex CMSs. Therefore, a method of NSSA of CMS based on semiquantitative information is proposed in this paper. First, through in-depth analysis of the mechanism of CMS network security indicators, reasonable indicators are selected, the weights of each evaluation indicator are calculated, and a CMS network security status evaluation framework including three-level indicators is built. Second, the ER algorithm is used to select the field management layer and the field equipment layer to evaluate the network security status of the CMS, and various uncertain information, including quantitative data and qualitative knowledge, is fused. An example is given to illustrate the detailed steps of the integration of indicator data by the ER algorithm. Third, the evaluation results of the network security status of the CMS are obtained by using ER fusion of the evaluation results of the 1 st level security indicators, and the security level is determined. Finally, through experimental verification, the proposed method is more accurate and can obtain more realistic evaluation results. In future studies, the evaluation method should be further optimized to improve the accuracy.
The ER-based network security state assessment method of CMS proposed in this paper has potential engineering application value and can provide an effective way to solve the network security assessment problem of complex dynamic systems. However, this method also has limitations. First, in practice, the reliability of data acquired by sensors cannot be guaranteed due to the existence of uncertainties, which will affect the rationality of the evaluation results. Second, if the experts are inexperienced, the accuracy of the evaluation results will be affected. Finally, the environment of CMS is very complex, and it will inevitably be interfered by real environmental noise and external environmental factors, which will also affect the accuracy of security state acquisition. Therefore, in future studies, it is necessary to further consider the reliability of sensor data when using the ER method for data fusion, and further optimize the proposed method to improve the accuracy of the evaluation results.
GUOHUI ZHOU received the Ph.D. degree from the Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun, China. He has published more than 20 articles in journals. His current research interests include artificial intelligence, pattern recognition, and embedded systems. He