An Evaluation System for HVDC Protection Systems by a Novel Indicator Framework and a Self-Learning Combination Method

High voltage direct current (HVDC) is expected to bring forth large capacity, long transmission distance, and asynchronous grid interconnection. To quantitatively analyze the protection systems of HVDC, an evaluation system is proposed with a novel indicator framework and an innovative weighting method for the assessment of HVDC operating status. The novel indicator framework includes 31 indicators from the perspectives of reliability, fault monitoring, operational maintenance, control efficiency, and system redundancy. A self-learning interval analytic hierarchical process is used to decide the weights of the indicators based on the maximum entropy method. The optimal subjective weights of the indicators can be obtained by the self-learning process, considering not only the fuzziness of single expert scoring but also the difference between experts’ weights. A real HVDC project in Hubei province, China, was studied to verify the effectiveness of the proposed evaluation system.


I. INTRODUCTION
The ultra-high voltage (UHV) and high voltage direct current (HVDC) have attracted more and more attention, which have the characteristics of large capacity, long transmission distance, and asynchronous grid interconnection. As an essential guarantee for the stable operation of the HVDC, direct current (DC) relay protection systems can not only monitor various operational parameters and states of power systems in real-time but also detect and handle abnormal conditions promptly. The development of power electronics, communication technology, and information technology are also imposing higher performance requirements on the HVDC and DC protection functions. Converter stations generally need to set up regular (e.g., once every six months, once a The associate editor coordinating the review of this manuscript and approving it for publication was Junjian Qi . month) or dynamic (e.g., online or irregular checks) evaluation of the HVDC protection system to improve the operation efficiency and reliability.
The core of the HVDC protection system evaluation is the determination of HVDC protection equipment's operation status. However, current HVDC protection system evaluation focuses on operation management, such as punching cards and data reports [1]. Limited data analysis, simple indicator check, and subjective judgments make it challenging to pinpoint the HVDC protection equipment's operation problems. The insulation systems of the equipment driven by power electronics are stressed by high-frequency voltages with harmonics from rectifiers and inverters. High-frequency voltages will accelerate insulation aging with many challenges to insulation systems [2]. Without a proper evaluation method of protection equipment, maintenance personnel may not detect the insulation risk caused by the insulation aging in time. VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ Besides, with the increasing complexity of the power network, the regular maintenance mechanism will bring tremendous resource waste and system risks [3]. By accurately evaluating the HVDC protection equipment status and formulating dynamic maintenance schemes based on the equipment status, the risk of improper maintenance due to the equipment status decline can be reduced, while under-or over-maintenance can be effectively avoided. In addition, the HVDC protection schemes adopted by Chinese HVDC converter stations are from different manufacturers, resulting in considerable differences in their effectiveness. The analysis and determination of the optimal protection scheme require a quantitative analysis of real cases. Therefore, an appropriate evaluation model is crucial for HVDC protection systems. Compared with those of the alternating current (AC) power system, the current evaluation indicators of the HVDC protection system are relatively scarce, resulting in a lack of research in the HVDC protection system evaluation field. Most adopted HVDC protection indicators are a rough summary of the report data, and only involve a specific performance such as reliability. The evaluation results only reflect the managers' workload, which lacks the consideration of the HVDC equipment status, operational maintenance, control efficiency, and system redundancy. Tu et al. [4] proposed some indicators based on droop control to evaluate the operating states of flexible DC distribution networks. To improve regional power grids with a high proportion of renewable energy, Wu et al. [5] proposed some evaluation indexes based on the security and stability of power grids. Rakhra et al. [6] demonstrated the impact of energy storage systems on the performance of existing DC protection systems and identified more suitable protection approaches to minimize the effects of protection blinding. MacIver et al. [7] presented a study on the reliability of different offshore grid design options to connect offshore wind power to shore. Mohan and Vittal [8] presented a performance evaluation of distance relay in the presence of voltage source converters-based HVDC systems. Existing literature has focused on the performance of either single electrical equipment or system operation [9]- [11], lacking multilevel and multidimensional evaluation of the overall HVDC protection system. At present, the field of HVDC protection urgently needs a comprehensive evaluation indicator framework that is highly compatible with the HVDC protection system and helps maintenance personnel understand the operating status of the equipment.
The weighting method is a vital part of the entire evaluation model. The combination of a reasonable weighting method and a comprehensive indicator framework is the basis for accurately reflecting the performance of the overall HVDC protection system. Evaluation methods are roughly divided into subjective evaluation methods and objective evaluation methods. The general subjective evaluation method focuses on the ambiguity of expert scoring, and the objective evaluation method focuses on data samples. To accurately evaluate the HVDC protection system operating status, this paper innovatively proposes the SLIAHP method.
Currently, the primary evaluation methods used in power systems include Analytic Hierarchical Process (AHP) [12], fuzzy AHP (FAHP) [13], [14], the entropy weight (EW) [15]- [18], and Delphi methods [19], among others. To adapt to the flexible systems with a large number of uncertainties, interval arithmetic (IA) can effectively deal with uncertain information. Based on IA, many related algorithms have been developed, including fuzzy interval [20], interval affine arithmetic [21], prediction intervals model [22]- [24], and interval optimization algorithms [25], which have been applied in various aspects of power systems [26]- [30]. Considering the uncertainty of expert scoring and user data, a hybrid interval AHP (IAHP) method was proposed for electricity user evaluation [31]. In the actual equipment operation process, the deterioration degree of each indicator is unequal, and thus it is particularly significant to modify the initial weight accordingly based on the change of the different indicator values. Mao et al. [32] proposed a variable weighting method for the comprehensive evaluation of transformer operation status, which adopted a linear weighting method to merge the subjective weighting method and the objective weighting method, by changing the subjective preference coefficient to achieve variable weights. Although the interval algorithm is mature, the weights are permanently determined by one expert scoring, which are challenging to adapt to multiple scenarios. In this paper, the above work is expanded, and a new self-learning interval AHP (SLIAHP) weighting method is innovatively proposed for the HVDC protection system evaluation with multilevel and multidimensional designs. The weighting method incorporates the opinions of multiple experts through self-learning, avoids a simple weighted average of different expert scoring, and realizes continuous weight updating through multiple scoring. The major contributions of this work are highlighted as follows: (1) A comprehensive evaluation framework of the HVDC protection system has been proposed for the evaluation. The proposed framework integrates the workflow of the evaluation system, structure of the HVDC protection systems, novel indicator framework, SLIAHP method, processing and decision-making systems, and communication systems to interpret the operation status of HVDC protection equipment.
(2) Based on the in-depth analysis of the HVDC protection system structure and understanding of the requirements of the HVDC protection system, an innovative evaluation indicator framework is proposed to evaluate the effectiveness of the HVDC protection systems from five different perspectives, which can accurately simulate the operation status of the HVDC protection equipment and discover the potential hazards of HVDC protection systems.
(3) A new self-learning weighting method is proposed considering both uncertainty and ambiguity, which reasonably integrates expertise from multiple fields and refines the weights through a self-learning process. The irrationality of averaging expert scores in the subjective weighting method is eliminated, and the dynamic update of the weights is implemented.
The rest of the paper is organized as follows. In Section II, the evaluation framework of the HVDC protection system is introduced. Section III describes the indicators framework for evaluating the operating states of the HVDC protection system. Section IV presents the SLIAHP weighting method for evaluating the operating state of the HVDC protection system. A case study on a real HVDC converter station in Hubei province, China, is presented in Section V. Finally, Section VI concludes the paper.

II. EVALUATION FRAMEWORK OF THE HVDC PROTECTION SYSTEM A. THE STRUCTURE OF THE HVDC PROTECTION SYSTEMS
To guarantee the overall operating performance of the HVDC system, the Chinese HVDC protection system structure is highly complicated. This paper mainly studies the HVDC protection system of converter stations, which can be classified into the following categories: MACH2, DCC800, HCM3000, HCM200, PCS-9550, among others. Based on the measurement-protection-exit structure, the HVDC protection system mainly includes measurement equipment, measurement interface equipment, HVDC protection equipment, 2-out-of-3 equipment, as well as trip equipment and secondary circuits. By integrating the equipment above, HVDC protection systems can offer pole protection (including bipolar protection), converter group protection, converter transform protection base on the electrical quantities, AC filter protection, and DC filter protection.
To ensure the reliable operation of the HVDC system and find the optimal protection scheme, it is necessary to establish a comprehensive evaluation system with the novel design of the indicator framework and weighting method. Figure 1 illustrates the overall evaluation framework with the core technologies and workflow of the proposed model.

B. THE PROBLEMS IN EVALUATING HVDC PROTECTION SYSTEMS
The problems in the evaluation of HVDC protection systems at present are summarized as follows: 1) The HVDC protection system has an intricate structure and multiple functions, but the current studies lack a comprehensive view of the HVDC protection system. It is not trivial to systematically evaluate the operating state without an HVDC protection system structure, and the indicator framework established so far cannot cover all the features of the HVDC protection system.
2) With the increasing complexity of the power grid structure across regions or provinces, the design, construction, and maintenance of the HVDC protection systems need to be continuously improved and optimized. Accordingly, the evaluation criteria for the HVDC protection system operating status also requires constant optimization and improvement in time.
3) Due to the different HVDC protection schemes adopted by converter stations, configuration and operating modes of different converter stations will not be uniform. To compare the operating states of HVDC protection systems at different converter stations, the evaluation system also requires high compatibility. 4) Compared with the AC power grids, the amount of the available operating data of the HVDC protection systems is relatively small. It is necessary to extract the operating data universally for all the converter stations and dig deep to obtain the implications of the collected data.

III. INDICATORS FRAMEWORK FOR THE HVDC PROTECTION SYSTEMS
To ensure that selected evaluation indicators are representative and necessary, the indicator selection should be based on the following principles: 1) Practicality. The selected indicators should have practical significance, be convenient for data collection and calculation, and make the indicator framework as refined as possible to ensure the objective and comprehensive evaluation results 2) Authenticity. The indicator framework's construction should be based on objective monitoring information and calculated data, reflecting the actual operating conditions and indicator relationships.
3) Rationality. The selected indicators should be typical, and the results have absolute reliability even when the number of indicators is reduced.
Based on the proposed indicator selection principle and the objective of the HVDC protection system, this paper firstly evaluates the HVDC protection system from the following five different perspectives, including reliability, fault monitoring, operational maintenance, control efficiency, and system redundancy. Except for some indicators, the remaining evaluation indicators presented in this paper are all innovatively proposed for the HVDC protection system and first applied to the HVDC protection system's operating status evaluation. To ensure that the proposed evaluation indicators can be highly compatible with the HVDC protection systems, most of the innovative indicators are defined according to the HVDC protection systems' characteristics and requirements. The multi-dimensional evaluation indicators framework is shown in Figure 2. The source and originality of the proposed indicator are shown in Table 1.

A. RELIABILITY
The reliability of the HVDC protection system operating status is shown in Figure 3. Reliability is one of the crucial factors that determine the quality of power supply to consumers [33], [34], and the reliable power supply of the HVDC system is studied in Section III.A. Its key indicators are introduced in the following subsections:

1) ENERGY UNAVAILABILITY A 1
The energy unavailability A 1 [35] refers to the reduction of delivering energy capacity due to outages or degraded operations within the statistical time frame. It represents VOLUME 8, 2020  the converter station's reliability and appropriately reflects the operating effectiveness of the HVDC protection system, which can be expressed as where N out represents the number of outage events, T ac−i represents the actual duration of the outage event i, P left−i represents the residual capacity in the outage event i, P e represents the rated capacity, and T se represents the statistical time frame.

2) ENERGY AVAILABILITY A 2
The energy availability A 2 [35] refers to the ability of delivering energy for the converter station within the statistical time frame, which directly reflects the operating effectiveness of the HVDC protection system. The energy availability A 2 can be defined as:

3) MONOPOLAR FORCED OUTAGE RATE A 3
The monopolar forced outage reflects the HVDC system's reliability and the operating performance of the HVDC protection system. The monopolar forced outage rate A 3 [35] represents the number of monopolar forced outage in the converter station within the statistical time frame, which can be expressed as: where N sin−out is the number of monopolar forced outages within the statistical time frame, and N po is the number of the HVDC pole pairs connected to the converter station.

4) BIPOLAR FORCED OUTAGE RATE A 4
The bipolar forced outage is a more severe situation than the monopolar forced outage. The bipolar forced outage rate A 4 [35] represents the number of bipolar forced outages in the converter station within the statistical time frame, which is calculated by: where N dou−out is the number of bipolar forced outages within the statistical time frame, and N po is the number of HVDC pole pairs connected to the converter station.

5) CONNECTIVITY A 5
The communication channel between the converter stations should be fast and reliable to transmit equipment status information. When the converter station faces outage due to a fault or loses the HVDC voltage control capability due to overloading, the HVDC protection system can adaptively select the takeover strategy through an inter-station communication channel to achieve automatic HVDC voltage control transfer. The connectivity A 5 is expressed as the number of the normal communication channel with other converter stations within the statistical time frame.

B. FAULT MONITORING
The fault monitoring of the HVDC protection system operating status has seven indicators as shown in Figure 4. The HVDC protection system is divided into seven parts based on the overall structure of measurement-protection-port, and the failure rate of each essential part is defined as an indicator. The source of the fault is monitored in Section III.B. Its key indicators are introduced in the following subsections:

1) MEASURING EQUIPMENT FAILURE RATE B 1
Essential for detecting and analyzing faults in HVDC protection systems, the measuring equipment consists of current transformers, potential transformers, zero-flux transformers, and conventional transformers. The measuring equipment failure rate B 1 determines the normal operation of the HVDC VOLUME 8, 2020 protection system, which is calculated by: where N me−er represents the number of measurement equipment failures within the statistical time frame, and N me represents the total number of measurement equipment.

2) MEASUREMENT INTERFACE EQUIPMENT FAILURE RATE B 2
To maintain the communication between the measurement equipment and the HVDC protection system, the measurement interface equipment consists of processor boards, switching value interface boards, analog quantity interface boards, communication boards, power modules, and chassis backplanes. The measurement interface equipment failure rate B 2 directly affects the real-time performance of the converter station, which is expressed as where N com−er represents the number of measurement interface equipment failures within the statistical time frame, and N com represents the total number of measurement interface equipment.

3) HOST-TYPE PROTECTION EQUIPMENT FAILURE RATE B 3
As an important part of the HVDC protection equipment, host-type protection equipment consists of industrial computers and distributed I/O systems. The host-type protection equipment failure rate B 3 affects the functional integrity of the HVDC protection systems, which is defined as: where N host−er represents the number of host-type protection equipment failures within the statistical time frame, and N host represents the total number of host-type protection equipment.

4) DEVICE-TYPE PROTECTION EQUIPMENT FAILURE RATE B 4
The device-type protection equipment is also essential to HVDC protection, which mainly consists of control protection equipment and distributed I/O systems. The device-type protection equipment failure rate B 4 affects the functional integrity of the HVDC protection systems, which is calculated by: where N de−er represents the number of failures of the devicetype protection equipment within the statistical time frame, and N de represents the total number of device-type protection equipment.

5) INDEPENDENT PROTECTION EQUIPMENT FAILURE RATE B 5
As a key part of the HVDC protection equipment, the structure of independent protection equipment is consistent with the AC protection equipment structure but without a distributed I/O system. The independent protection equipment failure rate B 5 affects the functional integrity of the HVDC protection systems, which is expressed as: where N ind−er represents the number of independent protection equipment failures within the statistical time frame, and N ind represents the total number of independent protection equipment.

6) 2-OUT-OF-3 EQUIPMENT FAILURE RATE B 6
2-out-of-3 equipment refers to a device that performs 2-out-of-3 logic judgment [37], which consists of processor boards, input and output boards, power boards, communication boards, and independent modules. The 2-out-of-3 equipment failure rate B 6 affects the functional integrity of the HVDC protection systems, which is defined as: where N 32−er represents the number of 2-out-of-3 equipment failures within the statistical time frame, and N 32 represents the total number of the 2-out-of-3 equipment.

7) TRIP EQUIPMENT AND SECONDARY CIRCUITS FAILURE RATE B 7
The trip equipment and secondary circuits are the basic components of the relay protection, which include switch operation boxes, cable circuits, optical fiber circuits, communication channels, and accessories (such as optocoupler and relay), among others. The trip equipment and secondary circuits failure rate B 7 affects the reliable operation of the HVDC protection system, which can be expressed as: where N br−er represents the number of trip equipment and secondary circuits failures within the statistical time frame, and N br represents the total number of trip equipment and secondary circuits.

C. OPERATIONAL MAINTENANCE
The operational maintenance of the HVDC protection system operating status has five indicators, as shown in Figure 5. The operating effect and configuration environment of HVDC protection equipment are quantified in Section III.C. The key indicators are introduced in the following subsections: The high working efficiency of maintainers can effectively improve the operating stability of the converter station [37]. The mean recovery time C 1 objectively represents the working efficiency of the HVDC protection system maintainers, which is defined as the average recovery time of all the repaired faults within the statistical time frame. It can be expressed as: where T end−i represents the failure repair time of equipment i, and T start−i represents the failure occurrence time of equipment i.

2) FAULT RECOVERY RATE C 2
The fault recovery rate C 2 indicates the ratio of the recovered fault number to the total fault number within the statistical time frame, which directly reflects the HVDC protection system's operating effectiveness. Fault recovery rate C 2 is calculated by: where N fault−repair represents the number of faults repaired, and N fault represents the total number of faults.

3) PROTECTION SYSTEM EXIT NUMBER C 3
Protection system exits will disable all the HVDC protection functions [37]. The protection system exits number C 3 represents the number of protection system out of operation within the statistical time frame.

4) SELF-TEST COVERAGE C 4
The condition-based maintenance is in the mainstream of the inspection methods, which should reasonably cover all the HVDC protection equipment to ensure the stable operation of the converter station [37]. The self-test coverage C 4 refers to the ratio of the number of overhauled HVDC protection equipment to the total number of the HVDC protection equipment within the statistical time frame, which is expressed as: where N pte refers to the number of total HVDC protection equipment, and N over is the number of overhauled HVDC protection equipment.

5) DATA SHARING DEGREE C 5
Maintaining communication among protection equipment in the converter station can effectively improve the HVDC system redundancy [38]. The data sharing degree C 5 reflects the normal communication ability of all the protection equipment within the statistical time frame, which is expressed as: (15) where N cr is the number of all communication failures, T cr−i represents the duration of communication failure i, and N pte−i is the number of protection equipment with normal communication during communication failure i.

D. CONTROL EFFICIENCY
The control efficiency of the HVDC protection system operating status has seven indicators, as shown in Figure 6. Based on the power supply requirements of the power system, the control accuracy and effect of the HVDC protection system are quantified in Section III.D. Its key indicators are introduced in the following subsections:

1) PROTECTION CORRECT OPERATION RATE D 1
The protection correct operation rate D 1 reflects the veracity of HVDC protection system [39], which is expressed as: where N cor is the number of correct operations of the relay within the statistical time frame, and N op is the total number of relay operations within the statistical time frame.

2) PROTECTION MAL-OPERATION RATE D 2
The protection mal-operation rate D 2 describes the sensitivity and veracity of the HVDC protection system [39], which is defined as: where N error represents the number of mal-operations within the statistical time frame. VOLUME 8, 2020

3) OPERATION BALANCE D 3
The operation balance D 3 can be used to quantify the balance of different redundancy schemes, which is defined as the ratio of mal-operation number to the refused operation number. The closer D 3 is to 1, the better the refused operation prevents mal-operation performance. The operation balance D 3 is expressed as: where N ref represents the number of refused operations.

4) FIRING ANGLE CONTROL ACCURACY D 4
The defect of firing angle control accuracy can significantly affect the control of the HVDC protection system. Under normal circumstances, the firing angle control error is ± (0.01 to 0.25) • . The firing angle control accuracy D 4 is defined as the maximum relative error of firing angle control at the converter station within the statistical time frame, which can be expressed as: where D act−i is the actual firing angle at time i, and D e−i is the reference firing angle at time i.

5) DIRECT CURRENT CONTROL ACCURACY D 5
The direct current control is an important control method for the HVDC protection system. Under normal circumstances, the direct current control error shall be ± (0.2 to 1) %. The direct current control accuracy D 5 is defined as the maximum relative error of direct current control at the converter station within the statistical time frame, which can be calculated by: where C act−i is the actual direct current at time i, and C e−i is the reference direct current at time i.

6) DIRECT POWER CONTROL ACCURACY D 6
The direct power control is an important control method for the HVDC protection system. Under normal circumstances, the direct power control error is ± (0.4 to 2) %. The direct power control accuracy D 6 is defined as the maximum relative error of direct power control at the converter station within the statistical time frame, which is defined as: where P act−i is the actual direct power at time i, and P e−i is the reference direct power at time i.

7) COMMUTATION SUCCESS RATE D 7
The commutation success rate D 7 is an indicator of the operating performance of converter stations, which indirectly reflects the control effectiveness of the HVDC protection system [39]. The commutation success rate D 7 can be expressed as: 22) where N cc represents the number of successful commutations within the statistical time frame, and N com represents the total commutations within the statistical time frame.

E. SYSTEM REDUNDANCY
The system redundancy of the HVDC protection system operating status has seven indicators, as shown in Figure 7. By connecting backup protection equipment, the redundant configuration of HVDC protection equipment can effectively reduce the risk of the HVDC protection system outage caused by a single equipment failure. The redundancy of HVDC protection equipment is quantified in Section III.E. The key indicators are introduced in the following subsections: At present, the control and protection systems of international converter stations are all double configuration or even triple configuration, including the operator control system, station control system, DC control system, DC protection system, dispatching automation system, and DC remote control system. From the overall design view, complete dualization requires that the I/O interface signals of the systems mentioned above should be configured in double configuration, which should be matched with the design of the primary equipment. For the HVDC protection system of the converter station, the redundant structure of each project is different [36]. The control and protection system must ensure its high reliability, which requires a redundant configuration. The choice of the redundancy mode directly affects its performance.

1) HOST-TYPE PROTECTION EQUIPMENT REDUNDANCY E 1
The host-type protection equipment redundancy E 1 represents the proportion of host-type protection equipment that meets the double configuration, triple configuration, or 2-outof-3 principle, which is expressed as where N host−red represents the number of host-type protection equipment that meets the double configuration, triple configuration, or 2-out-of-3 principle, and N host represents the total number of host-type protection equipment.

2) SENSOR REDUNDANCY E 2
The sensor redundancy E 2 represents the proportion of sensors that meet the double configuration, triple configuration, or 2-out-of-3 principle. The sensor redundancy is calculated by where N sensor−red represents the number of sensors that meet the double configuration, triple configuration, or 2-out-of-3 principle, and N sensor represents the total number of sensors.

3) AC FILTER REDUNDANCY E 3
The AC filter redundancy E 3 represents the proportion of AC filters that meet the double configuration, triple configuration, or 2-out-of-3 principle. The AC filter redundancy can be given as: where N fil−red represents the number of AC filters that meet the double configuration, triple configuration, or 2-out-of-3 principle, and N sensor represents the total number of AC filters.

4) AUXILIARY POWER REDUNDANCY E 4
The auxiliary power redundancy E 4 represents the proportion of auxiliary power sources that meet the double configuration, triple configuration, or 2-out-of-3 principle. The auxiliary power redundancy E 4 is expressed as: where N pow−red represents the number of auxiliary power sources that meet the double configuration, triple configuration, or 2-out-of-3 principle; N pow represents the total number of auxiliary power sources.

5) SWITCH REDUNDANCY E 5
The switch redundancy E 5 represents the proportion of switches that meet the double configuration, triple configuration, or 2-out-of-3 principle. The switch redundancy can be calculated by: where N swi−red represents the number of switches that meet the double configuration, triple configuration, or 2-out-of-3 principle; N swi represents the total number of switches.

6) DC LINE REDUNDANCY E 6
The DC line redundancy E 6 represents the proportion of DC lines that meet the double configuration, triple configuration, or 2-out-of-3 principle. The DC line redundancy is expressed as: where N line−red represents the number of DC lines that meet the double configuration, triple configuration, or 2-out-of-3 principle; N line represents the total number of DC lines.

7) CONVERTER VALVE SET REDUNDANCY E 7
The converter valve set redundancy E 7 represents the proportion of converter valve sets that meet the double configuration, triple configuration, or 2-out-of-3 principle. The converter valve set is defined as: where N fz−red represents the number of converter valve sets that meet the double configuration, triple configuration, or 2out-of-3 principle; N fz represents the total number of converter valve sets.

IV. THE SLIAHP METHOD
The subjective weighting method is a commonly-used method in power system evaluation models, which can incorporate the subjective experience of experts and enable the quantification of complex systems. As one of the subjective methods, the IAHP method effectively solves the uncertainties in the indicator calculation procedure. However, the following obstacles exist in practical applications: (1) Most work [11]- [14], [19], [31], [40], [41] only uses simple weighted averages to deal with the scores of different experts in the IAHP method, which cannot comprehensively consider the discrepancy and adaptability of expert opinions in various fields.
(2) The IAHP method is a weighting method based entirely on expert experience, which inevitably makes the results subjectively biased. The lack of objectivity may lead to limitations in the evaluation results.
(3) Different experts, application scenarios, and scores may bring specific differences to the weighting results. On-site, the evaluation of the HVDC protection system is mostly periodic or dynamic. If an expert scoring permanently determines the weights, the subsequent evaluation may no longer be applicable. The weight update by the IAHP method must start from the beginning, and the tedious process is not suitable for evaluating the actual HVDC protection system.
(4) The interval weights obtained by the IAHP method increase the workload of on-site managers, and the results of similar intervals are difficult to compare the advantages and disadvantages of the HVDC protection system. Therefore, the evaluation weights are best converted into constants.
At present, the mainstream evaluation methods cannot achieve variable weights, dynamic fusion of expert opinions, VOLUME 8, 2020 the fusion of subjective and objective theory, and ambiguity analysis. The SLIHAP method proposed in this paper solves the above problems well and realizes the dynamic evaluation of the HVDC protection systems. To make the evaluation results consider the subjective ambiguity of experts and contain subjective and objective theory, we constructed a new weighting method based on the IAHP method and the maximum entropy criterion (MEC) [42]. Since the traditional linear weighting method can only simply combine the weighting results of the two weighting methods in a specific ratio, it is impossible to form a new evaluation method by introducing objective theory into the subjective evaluation method. Taking the MEC as the optimization goal, we chose the Simulated annealing algorithm (SAA) [43] to improve the IAHP method in a self-learning way. Therefore, variable weights and dynamic integration of expert opinions can be achieved. Subsequent case analysis proves that the proposed SLIAHP method is beneficial for engineering applications. Compared with other variable weighting methods in the form of linear weighting [32], the proposed method is not a simple combination of subjective methods and objective methods in a specific ratio. Instead, it introduces objective theory into the subjective weighting method and constructs a new dynamic evaluation method by self-learning. The innovation of the proposed method is embodied in the following aspects: (1) The SLIAHP method employs the IAHP method to obtain the interval weight of all the indicators from experts in various fields, so that the weight results are integrated with the ambiguity of expert scoring.
(2) The SLIAHP method uses MEC to combine the opinions of experts in various fields. MEC minimizes the subjective assumptions caused by the lack of objective data in the evaluation results, thereby enhancing the objectivity of the weight.
(3) The SLIAHP method combines the experts' opinions in various fields, and obtains the indicator weights through self-learning based on the SAA. The self-learning process avoids a simple weighted average of different expert scoring and realizes dynamic evaluation with more practical application value.
(4) The evaluation weights are transformed from the interval value to a constant, convenient for practical engineering applications.
(5) At present, the SLIAHP method is first proposed in this paper and has not been applied to other research fields.
The flowchart of the SLIAHP method is shown in Figure 8, and the specific implementation steps are as follows.

A. THE IAHP METHOD
The IAHP analysis has been described in detail in the previous works [31], and its major procedure is summarized as follows. Step

1: Interval Judgment Matrix [A] Derivation:
According to the indicator framework for the HVDC protection system, a total of N experts are invited to distinguish the importance of all indicators at the different indicator where n denotes the total number of the HVDC protection indicators in the selected indicator layer. [a ij ] represents the importance difference of indicator i compared to indicator j, which is determined by: where i = 1, 2, . . ., n and j = 1, 2, . . . , n. a ij is the midpoint of the interval [a ij ], which determines the average importance difference between the indicator i and indicator j. The interval width µ is provided by an expert according to the uncertainty and vagueness.
Step 2: Interval Proportional Scale Derivation: The specific value of a ij is determined in this step. When the indicator i is thought more important than the indicator j, it should satisfy [a ij ] ≥ 1, i = j. Conversely, if the indicator j is considered to be more important than the indicator i, which should satisfy 1/[a ij ] ≥ 1, i = j. Different values of [a ij ] represent different degrees of importance. Step

3: Eigenvector Derivation
The largest eigenvalue [λ max ] and the corresponding eigenvector [P] of the judgment matrix [A] is calculated by the power method, which satisfy: Step 4: Consistency Check (CR): CR is used to denote the value of the relative consistency test. If CR of [A] is less than 0.1, [A] consequently passes the consistency test and can be applied to the next step. Consistency check is defined as: where the freedom index RI takes values from Table 2.

Step 5: Interval Eigenvector [P j ] of Each Expert:
The eigenvector [P] is extracted as a weight eigenvector [P j ] proposed by expert j, which contains the weight intervals of all HVDC protection indicators as described in model (34). Different experts have different opinions on the importance of the HVDC protection indicators. By repeating the above procedures, N interval eigenvectors [P j ], i = 1, 2, . . ., N , are obtained from N experts in various fields. Each interval eigenvector [P j ], which is defined as: where [ω j−i ] denotes the weight interval of indicator i presented by expert j.

B. MEC
The MEC is a criterion for selecting the statistical characteristics of random variables that best meets the objective situation [42]. When the MEC is introduced into the evaluation field, random variables can be replaced with weights. Information entropy H represents the elasticity of the evaluation system, and its increase will reduce system uncertainty and unreasonable risk of weight distribution. Due to the lack of objective parameters in the evaluation process, many unreasonable subjective assumptions have emerged. The principle of the MEC is to minimize the subjective assumptions of weights and maximize the information entropy H . Therefore, the most objective result should maximize the information entropy value under the known constraints.
To avoid a simple weighted average of expert scoring and transform the uncertain interval weight into a constant, a self-learning method combining the MEC and IAHP is proposed. Taking all interval eigenvector [P j ] presented by various experts as a constraint, the proposed method calculates the information entropy of different constant weight eigenvector ω. The weight eigenvector ω with the maximum information entropy can effectively reduce the subjective assumptions caused by the IAHP method and reasonably integrate each expert's opinion. Information entropy H can be expressed as: where H is the information entropy of the indicator weight, ω i is the weight of indicator i. Weight eigenvector ω refers to the weight vector in one case, which consists of n indicator weights. When a set of weights containing n variables is determined, the information entropy of the weights can be solved by substituting n weights into the model (35). The model (35) is introduced and solved in the steps 6 and 7 of section IV.C.

C. SELF-LEARNING OPTIMIZATION ALGORITHM
To eliminate the disadvantages of IAHP and make the results meet the MEC, the self-learning optimization algorithm based on the SAA is presented. In the process of self-learning, the training samples come from the expert judgment in various fields are employed to integrate multiple advice, and the MEC is taken as the objective function to reduce the subjective preferences and improve objective accuracy. As the key to achieving the above functions, the SAA is a random optimization algorithm based on Monte-Carlo iterative solution strategy [43], which has high iterative search efficiency and asymptotic convergence. The detailed steps of the SAA are presented below: Step 1: Calculate Initial Weight Eigenvector ω initial : If the IAHP method invites a total of N experts to score, N interval eigenvectors [P j ] will be obtained. The initial interval eigenvector [ω initial ] is the average of N interval eigenvectors [P j ] calculated by the IAHP method. Accordingly, the initial weight eigenvector ω initial consists of the midpoint of each weight interval of [ω initial ]. [P j ], [ω initial ], and ω initial are defined as: [ ω initial = ω 1 , ω 2 , ω 3 , . . . , ω n (38) where [ω j−i ] is the weight interval of indicator i according to the expert j, [ω i ] represents the initial weight interval of indicator i, and ω i denotes the initial weight value of indicator i. VOLUME 8, 2020 Step ]. Each training sample should satisfy the condition that the sum of n indicator weights is 1. The random weight sample W j can be expressed as: where ω j−i denotes the weight value of the indicator i in W j , which randomly takes values from weight interval [ω j−i ] according to the expert j.
Step 3: Obtain Random Disturbance : The initial weight eigenvalue ω initial is introduced into the self-learning iteration. To achieve dynamic weight updates, random disturbance during the iteration is determined by: Step 4: Update Weight Eigenvector ω: According to random disturbance in step 3, new weight eigenvector ω is updated, which is expressed as where d is a constant representing the step size of the random disturbance , which can be adjusted according to the actual convergence effect.
Step 5: Normalization: After normalizing the updated weight eigenvector ω, the normalized weight eigenvector ω * meet the following condition: where ω * i refers to the normalized weight value of indicator i.
Step 6: Calculate Information Entropy H(ω * ): According to the model (35), the information entropy H (ω * ) is calculated for the normalized weight eigenvector ω * , which is determined by Step 7: Metropolis Criterion: The MEC requires the weight results with maximum information entropy, so that the Metropolis criterion is adopted in the SAA. According to the model (45), if information entropy difference H > 0, ω * is accepted as a new initial weight eigenvector ω initial for the next iteration. Otherwise, with a probability of p( H ) in (46), ω * is accepted as a new initial weight eigenvector ω initial .
where k is the Boltzmann constant, T p represents the current temperature.
Step 8: Temperature Drop: The current temperature T p decreases at a certain rate, which is determined by where T 0 represents the initial temperature set before the iteration begins, and p refers to the current number of iterations.
Step 9: Terminating Condition: The self-learning process can be terminated when the weight information entropy reaches a maximum. If H ≤ 0 for h consecutive times, it is considered that the best effect of self-learning is reached in this iteration, the current initial weight eigenvector ω initial can be regarded as the final weight result, and the self-learning process ends; otherwise, the selflearning process returns to step 2 to regenerate training random samples for the next iteration. The breaking coefficient h can be set according to the acceptable computation time. A larger h gives a higher possibility of searching for the optimal global solution, while the search process may take a longer time; otherwise, the search process will take a short time but may fall into local optima.

V. CASE STUDY A. DATA SETS
To verify the proposed evaluation system for HVDC protection systems' operating states, five HVDC converter stations are selected for the case study. The Gezhouba, Jiangling, Longquan, Yidu, and Tuanlin converter stations are all HVDC converter stations located in the Hubei province of China, forming the most advanced HVDC transmission center in China. Their commutation capacity, transmission capacity, and equipment technology have reached worldleading levels. Therefore, the local economy and sustainable energy have developed rapidly due to the HVDC transmission project mentioned above. The HVDC protection evaluation framework of Hubei Province is shown in Figure 9. Based on the management system of State Grid Corporation of China, the operating data of the five HVDC converter stations were collected, and the min-max normalization results are shown in Tables A1 to A3 of the Appendix. The dataset contains operating data of nine continuous periods between January 2015 and December 2019. Ten experienced electrical engineers are invited to score the importance of the evaluation indicators of the HVDC protection system, and the specific judgment matrices are shown in Tables A4 to A63.

B. EVALUATION RESULTS
The initial temperature of the SLIAHP method is set to 3000 • . The iteration number p is set to 2500. According to the data sets, the indicator data was calculated by the presented indicator framework, and the weights were calculated by the SLIAHP method.
The indicator value is the most intuitive expression of the protection performance of the converter stations. By observing and comparing the indicator values of different converter stations, managers can find and improve weak parts of the HVDC protection system. Figure 10 shows the comparison of the HVDC protection indicators among the five HVDC converter stations. The following conclusions can be drawn from the results shown in Figure 10: 1) The energy availability of Gezhouba and Tuanlin converter stations has remained at a high level over the five years, but Yidu and Jiangling converter stations' energy availability fluctuated significantly in the five years.

1) INDICATOR RESULTS
2) Yidu converter station experienced a large failure in January 2017, and thus the duration of the failure far exceeded the peak value of other converter stations in each statistical period, which caused the mean recovery time C 1 of Yidu converter station in the month to be close to 0. The operational benefit and stability of the converter station are severely affected by long time operation under the fault condition, so the troubleshooting efficiency of Yidu converter station should be improved.
3) Among the HVDC protection indicators, the five converter stations performed better on monopolar forced outage rate, bipolar forced outage rate, device-type protection equipment failure rate, 2-out-of-3 equipment failure rate, trip equipment and secondary circuits failure rate, and fault recovery rate. 4) During the five-year operating process, each converter station's indicator value in the same quarter is similar. In other words, the trend of the indicator value has a cyclical characteristic, indicating that the operation of the converter stations has been stable and mature.

2) EVALUATION RESULTS
The overall evaluation result is a general description of the HVDC protection performance of the converter station, which is convenient for managers to grasp the protection equipment status of the converter stations. By horizontally comparing the evaluation results of different converter stations, the equipment aging and potential risks of the HVDC protection system can be discovered quickly. Converter stations with poor comprehensive performance can make targeted rectifications based on their indicator values.
To obtain the evaluation scores, the weights calculated by the SLIAHP method are multiplied by the corresponding indicator values, and the ultimate evaluation scores are taken as the sum of the 31 products. After considering the opinions of experts in various fields, the evaluation results with the SLIAHP method are shown in Figure 11 and Table 3.
The following conclusions can be drawn: 1) The overall evaluation rankings of the HVDC protection systems approximately match the advanced level (Completion time) of the actual converter stations, proving the rationality of the evaluation system.
2) Due to the increase in failures caused by aging equipment, the overall evaluation scores of some converter stations declined year by year. However, the scores of Gezhouba converter station are relatively stable, which reflects the high maturity of operation strategy, scheduling and maintenance plans.
3) Yidu and Jiangling converter stations have the highest evaluation scores due to the construction time, complete protection functions, and mature management experience. The Longquan converter station has unsatisfactory evaluation scores, which requires further improvements in equipment maintenance and relay protection.
4) The average evaluation score of Yidu converter station is 0.795, which has the best performance among the five converter stations. However, Longquan converter station's average evaluation score is 0.668, which has the worst performance. The overall performance of the five converter stations in 2019 is lower than the historical average performance. It is necessary to formulate rectification schemes to improve the HVDC protection system based on the indicator results.

1) COMPARISON OF THE AHP, IAHP, AND SLIAHP METHOD
AHP is a subjective evaluation method that focuses on the subjective experience of decision-makers and the characteristics of the problem under investigation. Compared with the AHP [39], the IAHP [40] and SLIAHP method comprehensively considers the ambiguity and uncertainty of the problem under investigation, giving the evaluation system better compatibility. Figure 12 shows that the weights obtained by the AHP and SLIAHP method are all within the weight interval obtained by the IAHP method. The weight trends calculated by the three methods are consistent, which proves that the three weighting methods can appropriately reflect the expert experience. The weight distribution of the SLIAHP method is more uniform than that of the AHP method. By retaining the subjective understanding of experts and the uncertainty of the investigated problem, the SLIAHP method effectively reduces the difference between the maximum weight and the minimum weight, so that the meaning of each indicator can be fully utilized and reflected. For the IAHP method, managers can adjust the weight within the weight interval to achieve the same effects.
In summary, the IAHP and SLIAHP method are more suitable for the evaluation of HVDC protection systems than the AHP method.

2) COMPARISON OF THE IAHP METHOD AND THE SLIAHP METHOD
When the weight obtained by the IAHP method is taken as the midpoint of the interval, the information entropy curve of two weights is shown in Figure 13 and Table 4. With the increase of the iteration number p, the information entropy of the SLIAHP method continues to increase, and the objectivity of the weight results has been improved continually. When the iteration number p is more than 2500, the increasing speed of the information entropy slowed down significantly. Compared with the IAHP method, the information entropy of the SLIAHP method has better performance.
Some conclusions can be drawn regarding the SLIAHP method:  1) The SLIAHP method alleviates the weight distribution imbalance caused by similar subjective preferences of experts, improving the objectivity of the indicator weights.
2) The uncertain weight interval range in the IAHP method is converted into a definite value in the SLIAHP method, which helps to simplify the evaluation steps and facilitates the operation of the management staff.
3) Due to less information entropy, Evaluation irrationality caused by significant differences between different weight values is balanced in the SLIAHP method.
4) The SLIAHP method eliminates the irrationality of the simple average of expert scores in the IAHP method, reduces the subjective assumptions due to the lack of objective data, and makes the HVDC protection weights more suitable for practical engineering applications.

3) COMPARISON OF THE EW METHO AND THE SLIAHP METHOD
To further verify the correctness of the method, the objective weight calculated by the EW method is introduced into the comparison. The EW method is described explicitly in the Ref. [17], which employs the data in Tables A1 to A3.
As an objective measure of the amount of information in an event, the information entropy is adopted by the EW method and the SLIAHP method to improve the evaluation objectivity. However, there are differences in the way these methods handle the information entropy. For the EW method, the larger the information entropy of the indicator, the smaller the amount of information provided by the indicator, and the smaller the role played in the comprehensive evaluation index, so the weight given is smaller [17]. For the SLIAHP method, the larger the information entropy of the weight distribution, the less the unreasonable subjective assumptions of weights, the smaller the system uncertainty and unreasonable risk of weight distribution, and the stronger the objectivity of the evaluation results.
The weight comparison curve of the EW method and the SLIAHP method is shown in Figure 14, and the following conclusions can be drawn: 1) The EW method is an objective weighting method relying solely on operating data, while the SLIAHP method is a subjective weighting method oriented by objective theory. Therefore, the weight curves calculated by these two methods are quite different. The SLIAHP method eliminates the singularity of a single evaluation perspective and integrates the experts' experience to make the evaluation results more convincing.
2) The EW method has higher requirements on the scale of input data, but the amount of data in the case is challenging to make the EW method achieve the desired effect. Besides, the EW method determines the weight by observing the change of the indicator. There is no statistical fluctuation of some indicators (e.g., bipolar forced outage rate A 4 , 2-outof-3 equipment failure rate B 6 , trip equipment and secondary circuits failure rate B 7 , fault recovery rate C 2 ) in the statistical time frame, resulting in these indicator weights calculated by the EW method are all 0. However, specific faults may occur in other actual scenarios, causing these stable indicators to change suddenly. To systematically reflect the HVDC system's abnormal conditions, each indicator must be combined with the corresponding weight to reflect the status of the HVDC protection equipment status. The SLIAHP method does not rely on the sample size like the EW method, whose weight distribution is more reasonable.
Above all, the SLIAHP method is more suitable for evaluating HVDC protection systems than the IAHP method.

VI. CONCLUSION
This paper proposes a comprehensive evaluation system for the HVDC protection system operating state. The evaluation system defines an innovative multi-dimensional indicators framework with a SLIAHP method to evaluate the overall performance of the HVDC protection systems. The proposed evaluation indicator framework accurately describes HVDC protection systems' characteristics and provides a detailed interpretation of the HVDC protection performance. The SLI-AHP method combines the opinions of qualified experts in various fields, considers ambiguity of the expert opinions and operating data, and improves the objective rationality of the evaluation by maximizing the information entropy of weights. A case study of 5 real-world converter stations in Hubei Province, China, proves that the evaluation system can support the operation, management, and maintenance of the HVDC protection systems.
LEIJIAO GE (Member, IEEE) received the Ph.D. degree in electrical engineering from Tianjin University, Tianjin, China, in 2016. He is currently a Lecturer with the School of Electrical and Information Engineering, Tianjin University. His main research interests include situational awareness of smart distribution networks, cloud computing, and big data.
YUANLIANG LI is currently pursuing the M.S. degree in electrical engineering with Tianjin University, Tianjin, China. His main research interests include situational awareness of smart distribution networks and big data.