Measurement Platform for Latency Characterization of Wide Area Monitoring, Protection and Control Systems

Wide area monitoring, protection and control (WAMPAC) systems have emerged as a critical technology to improve the reliability, resilience, and stability of modern power grids. They are based on phasor measurement unit (PMU) technology and synchronized monitoring on a wide area. Since these systems are required to make rapid decisions and control actions on the grid, they are characterized by stringent time constraints. For this reason, the latency of WAMPAC systems needs to be appropriately assessed. Following this necessity, this article presents the design and implementation of a measurement platform that allows latency characterization of different types of WAMPAC systems in several operating conditions. The proposed WAMPAC Characterizer has been metrologically characterized through a WAMPAC Emulator and then used to measure the latency of a WAMPAC system based on an open-source platform frequently used by transmission system operators (TSOs) for the implementation of their PMU-based wide area systems.


I. INTRODUCTION
I N RECENT decades, monitoring of power transmission systems has undergone a significant upgrade with the introduction of wide area monitoring systems (WAMSs) based on phasor measurement unit (PMU) technology and their evolution toward wide area monitoring, protection and control (WAMPAC) systems.WAMPAC systems, based on advanced monitoring and communication technologies, offer an innovative approach for real-time (RT) synchronized monitoring, advanced protection and automated control on a wide area.The growing interest in WAMPAC systems is driven by the need to manage increasingly complex and interconnected electric grids, which are characterized by a high variability, due to the growing presence of distributed energy resources, the automation foreseen by smart grid paradigm and the continuous increase in energy demand.These factors have made the development of innovative solutions for monitoring and managing grids on a large scale essential.
WAMPAC systems involve, as the first element in their monitoring architecture, the PMU [1], which is capable of measuring the so-called voltage and current synchrophasors, instantaneous frequency and rate of change of frequency (ROCOF) [2], and the phasor data concentrator (PDC), which enables the reception and time alignment of measurements from PMUs distributed throughout the power grid [3].Measurements made by PMUs require a common and absolute time reference to be compared with each other and, for this reason, synchronization technologies that can provide high temporal accuracy (better than 1 µs [2]) are used.To this aim, PMUs are often equipped with a global positioning system (GPS) receiver or make use of packet-based synchronization, like that provided by precision time protocol (PTP), to tag each measurement with an absolute timestamp in the coordinated universal time (UTC) timescale.The measurements computed by PMUs are usually encapsulated in data packets compliant with the IEEE C37.118.2 standard [4].
PMUs play a crucial role in a wide range of WAMPAC applications, as the monitoring and control actions are based on synchrophasor, frequency, and ROCOF measurements.In [5], WAMPAC applications based on ROCOF estimation with three established state-of-the-art algorithms are analyzed, and their impact on the successful grid restoration is shown.In [6], several WAMPAC applications are described, including recording of dynamic events, RT system state determination, and phase-angle monitoring (PAM).PAM, for instance, is an application of significant relevance to WAMPAC systems, as it provides information about power transfer [7].In addition, it allows handling angular instability issues in the transmission network by activating RT protection functions to prevent propagation phenomena and thus improve resilience [8].As often happens with WAMPAC applications, there are still no standardized recommendations, e.g., regarding angular difference bounds for transmission grids, as pointed out in [8].Indeed, the thresholds considered by WAMPAC depend on the specific context and transmission system operator (TSO) expertise.
The other key component in the measurement chain of the WAMPAC system is the PDC.This element, initially conceived as a module responsible for receiving data from the PMUs, is now assuming an increasingly active role in control systems, as it is the first element to supervise a network portion in the synchrophasor hierarchy.Indeed, the IEEE C37.247-2019 PDC standard [3] recommends the ability of PDCs to implement control logic, such as event detection and classification, and examples of these techniques are described in [9].In [10], a PDC prototype capable of handling delay and accurately evaluating PMU data flow latencies is presented.In [11], the architecture and validation of a low-latency PDC are studied using different network infrastructures, e.g., with fiber-optic cables or with 4G LTE wireless technology.
In this context, one of the critical aspects in the design and implementation of WAMPAC systems is the total latency, i.e., the time delay between the occurrence of an event in the electric system and the execution of associated control actions.Latency can greatly affect overall system performance, with possible implications for the safety, reliability, and stability of the grid.Therefore, it is useful to evaluate the latency of each element within the monitoring and control chain.In this regard, it would be beneficial to propagate time properly across the entire network infrastructure [12].
Depending on the implemented algorithm, PMUs can introduce considerable delay into WAMPAC systems and, for this reason, the literature has focused on their latency characteristics and requirements.In [13], for instance, a low-latency (less than 10 ms) algorithm, suitable for control applications, is proposed.The characterization of PMU latency is analyzed in [14] and [15].
To reduce the delay caused by communication, the network topology must also be carefully evaluated [16], [17], and different communication technologies, such as supplementary wireless link, can be used to compensate temporary wired network failures [18].In [19], the experience of the Croatian Transmission System Operator (HOPS) with a PMU-based angle stability protection application is presented.The communication latency due to the different geographical locations of the PMUs installed in their network is discussed.
Focusing on the control chain, a characterization of the operational delays of control systems using the RT digital simulator (RTDS) with the hardware-in-the-loop (HIL) architecture is presented in [20].A similar approach is followed in several scientific works including [21] (using OPAL-RT) and [22], [23] (using RTDS), but, in this literature, no measurement tool is provided specifically for the latency of the WAMPAC system and no metrological assessment is reported.In [24], EPRI reports the results of experiments involving two ad hoc modified PMUs, an event and time monitor, and a PDC.Using different transport protocols, the delays of the event detection are computed with resolutions equal to the reporting interval.
From another perspective, WAMPAC systems, which act automatically on the power network, should also be secure and resilient against possible cyber-attacks.In this context, cybersecurity in WAMPAC involves several levels, from design to system vulnerability testing.A detailed discussion can be found in [25], and a series of technical recommendations are reported in [26].Focusing on specific examples, in [27], possible attacks on a WAMPAC infrastructure equipped with attack-resilient algorithms and a defense architecture are explored.In [28], [29], and [30], the impact of time synchronization attacks is experimentally evaluated, which is important since PMU synchronization error impacts on the accuracy of phase angle and derived measurements and also on the associated timestamps, thus affecting the PMU sending time and latency.
Given the importance of characterizing the delays within WAMPAC systems, in [31] the latency of a WAMPAC prototype of the Italian TSO was characterized through an emulator of two PMU transmission control protocol (TCP) data streams, with fixed reporting rate (RR) and packet size, and based on the 4-20-mA actuator technology.
Stimulated by these preliminary studies, this article presents the design and implementation of a characterizer of WAMPAC latency, i.e., an instrument to accurately measure latency in purely digital WAMPAC systems, which allows testing several PMUs at once, with different protocols, packet sizes, and RRs.A modular instrumentation platform by National Instruments (NI) equipped with RT hardware is used to prototype the proposed design.To carry out a metrological verification of the functionality and performance of the proposed instrument, a second device has been designed and implemented to act as an emulator of a WAMPAC system, which is programmable and able to mimic real WAMPAC features.Then, the experimental characterization of a WAMPAC system based on software tools commonly used in real WAMS/WAMPAC applications is illustrated and confirms the validity of the proposed measurement platform.

II. WIDE AREA MONITORING AND PROTECTION SYSTEMS
The considered WAMPAC architecture, shown in Fig. 1, consists of PMUs scattered throughout the territory, a WAMPAC core (referred to as WAMPAC server, WS from here on), and a module representing the control actuator.This is a common scheme and is used throughout this article for presentation purpose.Specific adaptations of such architecture can also be addressed with slight variations in the proposed approach.
PMUs are distributed in the power network (e.g., in 380-and 220-kV substations) and can be configured according to two compliance classes, Classes P and M, defined by the synchrophasor standard IEC/IEEE Std 60255-118-1 [2].Class P is used for applications that require measurements with short response times, and thus, it is expected to play a role especially for electric system protection, whereas Class M has less stringent timings and is preferable where better disturbance rejection and larger off-nominal ranges are more important.For this reason, the use of P-type algorithms is often considered preferable for WAMPAC systems, to reduce the so-called reporting latency introduced by PMUs, which mainly depends on the algorithm computation window.An interesting solution for WAMPAC systems would be to choose PMUs that meet the more stringent requirements of the two classes [32].Also crucial is the choice of the RR of the PMUs on which latency may depend.The RR commonly used by TSOs is 50 frames/s, which means a packet containing measurements every 20 ms [33].
These packets are typically received by an ad hoc PDCenabled device (the WS in the considered architecture) that time-aligns the packets and then, in the WAMPAC architecture, analyzes the incoming data and sends a control packet according to several possible implementations.For example, it may send a command only when the PMU measurements satisfy a certain WAMPAC policy.The packet sent by the WS can be of various types, such as a routable-generic object oriented substation event (R-GOOSE) message conforming to IEC 61850-90-5 [34], like in the WAMPAC system in [31], or an IEEE C37.118.2 packet as those sent by the PMUs.
The IEEE C37.118.2 standard specifies the various packet types or frames for RT communication between devices in a distributed measurement architecture.The standard outlines frames of the header, data, command, and configuration types.
The focus here will be on the data frames, which contain the measurements collected from the PMUs, as well as command frames.Command frames are primarily used to initiate communication between PDCs and PMUs or to request information from the devices.In a pure monitoring context like WAMS, the role of the command frame is considered limited to the basic functionality defined in the IEEE C37.118.2 protocol.However, the standard defines seven standard commands (turn on transmission, turn off, send configuration frames, etc.), with some bits reserved for user designation.Widearea applications that rely on synchronized measurements may thus use updated versions of the command frames for specific control applications.
An updated IEEE C37.118.2 command frame with a specific command for communication between WS and control actuator is considered in this article as an example of possible implementation.Such command frame allows a PDC to evolve into an active device for power system control.This message can be used in many applications as the activation signal for the actuator of the control (see Fig. 1).Otherwise, the content of the control packet can be transformed by adapters/gateways into various analog control signals if a WAMPAC system requires it (e.g., current signals in the 4-20-mA range as in [31]).The control actuator, based on the PMU measurements and the policies implemented by the WS, allows physical intervention in the system according to the required strategy.
The total latency L T of the system, schematized in Fig. 1, is given by with where the below can be seen.
1) PMU reporting latency L PMU in (1) is the time interval between the measurement time as indicated by the data timestamp and the instant when the data become available at the PMU output [2].2) WAMPAC latency L WAMPAC is defined in (2) as the sum of the following terms.a) PMU communication latency L D is the time needed for the network to transmit the IEEE C37.118.2 data frame from the PMU to the WS.b) WS latency L WS is the time required for the WS to process data from the PMUs and send a control/command message.c) WS communication latency L C is the delay of the control/command message sent by the WS over the network.
The latency of the PMU is not taken into account in L WAMPAC , as it depends mainly on the instrument, and the objective of latency characterization is twofold: find the total latency and find the latency introduced by the WAMPAC system regardless of the chosen commercial PMUs or their configuration.It is worth recalling that L PMU depends on the performance class and RR considered, and its maximum value is recommended in [2].For this reason, and since L PMU is usually measured in advance by the manufacturer or laboratory characterization, here the focus is on L WAMPAC .As it will be described in what follows, the designed characterization platform uses the IEEE C37.118.2PMU packet output time (referred to as PMU sending time in the following) for the calculation of L WAMPAC .
It is worth noting that in a distributed PMU framework, i.e., in the presence of merging units or instrument transformers with digital output, the contributions given by the latency of the digitizer and the communication latency of the packets carrying the sampled values sent to the PMU [35] affect L PMU .Therefore, these delays have to be characterized to understand their weight.
L D and L C also include the latency caused by devices, such as routers, in the communication network.Moreover, L D may be different between PMUs because they are located at different nodes in the network.
L WS can in turn be divided into two contributions, according to needed WS functionality operation [3].
1) The "WS wait time" starts when the first PMU message with a given timestamp arrives and ends when the last message is received or the configured maximum wait time expires, after which the partial dataset is handled.
The difference between the end of such wait time and the arrival time of the last on-time PMU packet contributes to L WS .2) The "WS processing time," usually quite smaller than the first, represents the time for the WS to process and complete the output stream.In WAMPAC applications, this interval includes WAMPAC policy evaluation.
As a general consideration, if the architecture is more complex than that in Fig. 1, e.g., if several PDCs are present in the hierarchy, the latency of each device and the communication latency of their sent packets are also considered in L WAMPAC .
Furthermore, if the message sent by the WS needs to be transformed into an analog signal, the latency given by the transducer device, which is usually negligible with respect to other contributions, should also be considered.

III. PROPOSED LATENCY CHARACTERIZATION SYSTEM
To evaluate the latency of a generic WAMPAC system, a "WAMPAC Characterizer" (also briefly "Characterizer" from here on) has been designed and implemented to emulate the behavior of different PMUs.Indeed, the Characterizer allows sending several packets compliant with the IEEE C37.118.2 standard as if they were originated from a set of PMUs.In each PMU stream, packets are sent every T RR instants, i.e., according to the configured PMU RR.On the other hand, the Characterizer allows receiving control packets from the WAMPAC system, thus closing the control chain and allowing latency evaluation.As mentioned above, control packets are here assumed, for the sake of presentation clarity, to be formatted according to IEEE C37.118.2, which is a realistic choice in many applications.This choice goes in the direction of industrial best practice [26] and the typical scenario is that of TSO private or virtual private communication networks.The Characterizer matches the PMU communication capabilities and, in this regard, if additional security reinforcement approaches (e.g., security gateways) are used for PMU data, they can be applied straightforwardly also to the Characterizer's output.It is foreseeable that in future scenarios, when security protocols may be applied directly to PMU streaming [36], dedicated solutions can be used to extend the Characterizer's capabilities in this perspective.
The WAMPAC Characterizer, which will be described in detail in Section III-A, is an instrument devoted to WAMPAC latency measurement and thus needs in turn to be characterized to assess its performance before its application on the field.For this reason, a device called "WAMPAC Emulator" (or "Emulator") was also designed and implemented to physically emulate the behavior of a generic WAMPAC architecture with minimum additional latency.
Both the Characterizer and the Emulator allow configuring the number of PMUs, their RR, and the PMU data frame size so that different scenarios of WAMPAC systems can be tested.
Fig. 2 shows the setup used for the laboratory characterization tests, in which there is a direct connection between the Characterizer and the Emulator to measure the latency required by a WAMPAC chain that mimics a real architecture without the impact of communication latency due to network devices and external network traffic.Through characterization, which will be discussed in Section IV-B, an evaluation of the internal latency of the Characterizer can be established to provide an indication of the measurement error introduced by the instrument.Then, the Characterizer can be applied to a real WAMPAC scenario (see Section IV-C).

A. WAMPAC Characterizer
The WAMPAC Characterizer allows the emulation of a variable number of PMUs (during the test, this number was raised up to ten, but more PMUs can be emulated depending on the computational capability of the considered hardware platform) capable of generating IEEE C37.118.2 packets with different packet sizes.User datagram protocol (UDP) or TCP can be chosen for the transport layer protocol as common to commercial PMUs.The RR of PMUs is also user-modifiable, and an RR up to 200 frames/s can be configured.Once again, higher RRs can also be used depending on the computational power available.These features make the WAMPAC Characterizer more flexible for testing WAMPAC systems with different requirements compared with the solution in [31], which allowed emulation of only two PMUs with RR at 50 frames/s and TCP messages of fixed packet size.
To minimize the intrinsic latency of the Characterizer, which inevitably affects the measured WAMPAC latency, the emulated PMU packets are generated at UTC time instants corresponding to the PMU reporting instants (multiples of T RR and written as the packet timestamps).In this way, packet flows emulate the real behavior of PMUs without introducing PMU measurement latency (L PMU ≈ 0).Moreover, to increase the level of emulation of the PMUs, the Characterizer can implement a specific delay for the sent packets of each PMU.In this way, real cases in which the PMUs do not have the same communication or reporting latency can be simulated.
Whenever packets have to be sent, the sending time of each packet (t DS,i , where i represents the index of the emulated PMU) is obtained and written into the sent packet as an analog field.It allows having also the sending time available at the side of the receiver and not only the PMU timestamp (reporting time).Clearly, this option can be deactivated when the WAMPAC does not support the interpretation of this additional information, or the exact packet format of the real system needs to be used.After that, t DS,i of each PMU is stored in the system.
As mentioned above, the Characterizer is programmed to receive IEEE C37.118.2 packets from the WAMPAC system in response to PMU packets.Whenever a control packet is received, the current time (t CR ) is stored, and the difference between this arrival time and the largest timestamp of the corresponding PMU packets is calculated, thus computing the following latency: where denotes hereinafter the measured quantities, and N is the total number of emulated PMUs.
In addition, the Characterizer also computes the received control packet latency (WS communication latency) Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.defined as the difference between the WS packet arrival time t CR and the timestamp of the command frame, t CS .In this way, the Characterizer calculates the latency of the packets coming from the WS, estimating the communication latency and allowing to highlight its contribution to the WAMPAC latency.Fig. 3 shows the main functionalities of the Characterizer using a block diagram.The proposed instrument is designed to rely on RT hardware and software and on a platform equipped with field-programmable gate array (FPGA) to guarantee determinism in performance.These features are important to address timing and synchronization requirements and achieve high measurement accuracy as will be discussed in what follows.Configuration operations are performed on the parameters that can be set, such as the number of PMUs N , RR, packet payload size, protocol type, and input data file.The actual synchrophasor streams of the various emulated PMUs can be chosen by the user based on an auxiliary file of measured values (e.g., three-phase synchrophasor, frequency, and ROCOF values for a certain time interval, collected from the field or simulated).The measured values can be used cyclically to test WAMPAC response repeatedly during long tests.The Characterizer can also provide simple measurement streams generated internally.
The main functionalities are divided into two parallel RT cycles, disciplined by the internal clock with a frequency equal to 10 MHz.The top RT cycle allows the set of PMUs to be emulated in terms of data transmission and timestamping.Indeed, the first block ensures alignment with reporting time instants by calculating the wait time required for alignment according to UTC.Whenever a time reference is needed, as seen in Fig. 3, the program makes a software call to the FPGA, which provides UTC time.Indeed, the FPGA is synchronized using an external time source; in this article, a GPS receiver is considered.
Within a number N of parallel instances, the UTC time is obtained and, consequently, an IEEE C37.118.2 data frame packet is built and sent with the corresponding reporting timestamp and, as mentioned above, with the measured sending time t DS,i .The parallel instances are distributed efficiently according to the number of processors available in the platform adopted for implementation.At the output of the parallelized loop, a stop condition is evaluated: for instance, if the count of sent measurements is less than a set number (which depends on the RR, the length of the test, and the number of stream iterations), the RT loop is re-executed, otherwise the loop terminates.
The bottom RT loop allows the command or data frame packets of the WS to be received.The possibility of receiving data frame packets was also included to allow testing even WAMPAC systems based only on a PDC and thus on data frames.The first block is responsible for receiving and checking these packets.
When a packet is received, the current time is calculated and saved as t CR and immediately afterward the packet is decoded, extracting useful data such as t CS .After decoding, L WAMPAC and L C are computed and stored according to (3) and (4), respectively.Finally, a stop condition is checked.
The proposed Characterizer has additional features: it can be indeed configured to also be used as an instrument to measure L T , thus including PMU latency by checking the arrival time of the control packets with respect to the PMU timestamp.In this perspective, it allows testing mixed scenarios where the emulated PMUs and commercial PMUs coexist.L PMU can be set for each emulated PMU to test different monitoring configurations.
The Characterizer was implemented in a NI CompactRIO-9068 (NI cRIO-9068), a modular system equipped with the NI Linux RT operating system and provided with ARM Cortex-A9 dual-core processor with a maximum frequency of 667 MHz and 1-GB RAM.The NI cRIO-9068 was equipped with the NI 9467 module for GPS synchronization with a pulse per second (PPS) accuracy of ±100 ns.In this article, to reflect the behavior of digital WAMPAC systems, an IEEE C37.118.2 packet was chosen as the activation signal.Nonetheless, the modular and configurable hardware is ready to adapt the system to the needs of the specific WAMPAC system under test by changing the nature of the activation signal, for instance, using a current signal in the 4-20-mA range, or by changing the communication protocol, such as the IEC 61850-90-5 [34].
A specific version of the Characterizer's software adapted to generic personal computers (PCs) was also implemented to provide another tool for preliminary latency assessment to TSOs and to evaluate the performance differences when using general purpose (GP) instead of RT hardware.This version differs from that used in the RT platform in terms of timestamping, since, in the absence of the FPGA, a software call to the computer time is used.The PC clock is synchronized via network time protocol (NTP).For instance, in the tests of Section IV below, the synchronization is provided by a Meinberg Lantime M1000 GPS receiver within the same LAN of the PC.Due to the non-RT behavior, it is not possible to have a highly accurate time and PC clock offset may drift up to 10 ms from the reference.However, the detected drift has a negligible impact on WAMPAC latency measurements when no strict synchronization is required by WAMPAC operation, and it mainly affects communication latency measurements.For instance, if the Emulator is used, packet communication latencies to and from it are calculated based on timestamps from different timescales (internal computer clock and WS clock).

B. WAMPAC Emulator
The WAMPAC Emulator allows the emulation of fundamental WAMPAC functionalities to test the behavior of the Characterizer.Indeed, the Emulator implements the basic functions of the WS and can thus serve as a low-latency version of a real system.Therefore, the WAMPAC Emulator is capable of receiving the IEEE C37.118.2 packets from PMUs (both real and emulated) and can run in RT a WAMPAC logic.Indeed, a policy must be chosen to decide when the activation signal has to be sent (in an IEEE C37.118.2 command frame packet in this article) for controlling the grid.
An example of WAMPAC policy based on phase-angle measurements, which is also adopted in the tests of Section IV, is the following rule: where the activation signal is generated and the corresponding command frame is sent if the phase-angle difference between the phase-angle measurement ϕ i ref of a PMU i ref , chosen as reference, and at least one of the measured phase angles ϕ i included in the other PMU packets with the same timestamp is greater than a threshold δ ϕ .On the contrary, i.e., under normal conditions, the command packet is not sent.The simple PAM application considered here is inspired by that used by the Italian TSO in [31].Furthermore, this simple policy appears effective when some real cases, such as the incident that occurred at the Continental Europe Synchronous Area (CESA) on January 8, 2021, are considered [37].In that situation, shortly before the failure, the system was already operating close to the point of angular instability, with voltage phase-angle differences close to 90 • between Western and Eastern Europe [37].This event led to the separation of CESA into two subareas and demonstrated once again the importance of monitoring phase-angle differences between voltage phasors measured with PMUs at different locations, as they can be considered an early index of power transmission network stress [6], and thus may help in taking automatic countermeasures through purposely designed WAMPAC systems.
The Emulator is also capable of calculating the IEEE C37.118.2 packet latency L D,i for the incoming PMU streams defined as the difference between the arrival time of input packet t DR,i and the sending time t DS,i written into the same packet, i.e., In this way, the Emulator can analyze the behavior of the PMUs emulated by the WAMPAC Characterizer in more detail, providing additional insight.The Emulator also estimates its own processing latency L WE as L WE = t CS − max(t DR,1 , t DR,2 , . . ., t DR,i , . . ., t DR,N ).(7) L WE , as the WS processing time defined in Section II, represents the difference between the instant when the activation signal is sent (t CS ) and the arrival time of the last PMU packet among those with the same timestamp that trigger the policy.
Analogous to that illustrated for the Characterizer, the block diagram of the Emulator implementation is shown in Fig. 4. Also in this case, configuration operations are carried out on the parameters that can be set.In the same way as the Characterizer, the Emulator uses an RT cycle, regulated by the internal clock at 10 MHz, containing an internal cycle with N parallel instances distributed appropriately to the available processors.
Within the parallelized loop, PMU packets are received.The arrival time t DR,i is saved and the data frame is decoded, extracting t DS,i .After obtaining the required data for all the incoming PMUs, the WAMPAC policy is applied (voltage and current synchrophasors, frequency, and ROCOF can be considered according to the adopted strategy).If the policy condition is verified, a timestamp is recorded and the command frame is built.
The output packet was assembled according to the standard [4], using the bits of the CMD field reserved for user designation (identified with bits 15-12 set to 0 and bits 11-8 not equal to 0), to obtain a specific command for WAMPAC application.
Therefore, in accordance with the WAMPAC policy summarized in (5), this functionality has been used to send the control packet to activate the control strategy.
After the packet is sent, the WAMPAC Emulator's processing latency L WE and the PMU communication latencies L D,1 , L D,2 , . . ., L D,i , . . ., L D,N are calculated, according to ( 7) and ( 6), respectively.If the policy results in a false condition, only the PMUs' communication latencies are computed.Finally, the RT cycle is re-executed according to the stop condition.
For the implementation of the WAMPAC Emulator, another modular system was used, the NI cRIO-9039, provided with NI Linux RT operating system, Intel Atom quad-core processor with a maximum frequency of 1.91 GHz and 2-GB RAM, and equipped with the NI 9467 module for GPS synchronization.As in the WAMPAC Characterizer, the synchronization allows timestamping the packets arrival and departure with respect to UTC and consequently to calculate the aforementioned latencies in a coherent way.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.Picture of the used hardware and connections for laboratory characterization.
Fig. 5 shows the modular devices used for the test setup during laboratory characterization and highlights the involved connections.

IV. TESTS AND RESULTS
In this section, two selected test scenarios and the corresponding test setup are presented.Then, the most significant results are reported and discussed.

A. Test Scenarios and Setup
The first test scenario corresponds to the Characterizer's latency laboratory assessment involving the Emulator introduced in Section III and, in particular, in Section III-B.The test setup corresponds to Fig. 2. In the performed tests, PMU data correspond to synthetic synchrophasors generated directly by the WAMPAC Characterizer.The measured values were kept stable: for an emulated PMU assumed as reference, the phase angle was constantly equal to 0 • while for the other emulated PMUs, it is kept at 110 • , so that the phase-angle difference was always greater than δ ϕ = 100 • , which is considered as the event detection threshold.In this condition, the WAMPAC Emulator sends, for each set of packets from the emulated PMUs (each timestamp), a control packet.This allows characterizing the device in the worst stress condition, when it produces a latency measurement approximately every T RR .This choice also permits a statistical assessment of latency measurement characteristics within short time intervals and is thus adopted from here on for the tests.
The second test scenario is an experimental latency measurement test for a WAMPAC system based on the OpenPDC tool by Grid Protection Alliance (GPA) [38].This is an open-source software application intended as the core module in the WAMS and WAMPAC systems design.The OpenPDC is a data concentrator whose primary function is to time-align and collect the synchrophasor data provided by the PMUs.It enables the management of data from PMUs in the field by also allowing outbound streams to be sent, concentrating data from the various incoming packets.Due to its capabilities, this software has been used in various scientific projects and real-field implementations [39], [40].
The test architecture is shown in Fig. 6(a).The OpenPDC is hosted in a workstation equipped with two Intel Xeon Gold 6128 processors, 256-GB RAM, and Windows 10 OS.
To simulate various behaviors of a real network, a network emulator has also been used within the previously described architecture, thus modified as shown in Fig. 6(b).It is then possible to test applications in the presence of real network issues such as higher latencies, jitter, and packet loss conditions.Network emulation with a similar approach has been used in several scientific works [10], [41].The network emulator has been hosted in a workstation, with an Intel E8500 processor, 8-GB RAM, and a Linux operating system (Ubuntu 16.04 LTS), running the NetEm software [42].The adopted workstation is equipped with several Ethernet network interfaces; two of them have been configured in the transparent bridge mode, to influence the input and output data streams.As shown in Fig. 6(b), the network emulator has been connected between the WS and the Characterizer to simulate an additional latency reach the actuator.
An additional test setup, shown in Fig. 6(c), has been used to test the same WAMPAC architecture in a mixed scenario, i.e., with the Characterizer and real PMUs connected to the same OpenPDC.In this case, the aim is to measure L T , thus including the PMU reporting latency of a commercial PMU and the latency of PMUs emulated by the Characterizer.

B. Validation of the WAMPAC Characterizer
The first series of tests is devoted to the characterization of the metrological performance of the WAMPAC Characterizer in a controlled laboratory environment (see Section IV-A).The tests have been divided into short and long tests.
The Characterizer was initially verified with 15-min (shortterm) tests in which all the configuration parameters were varied to check the instrument operation for every possible combination.Specifically, the following options have been used.
For the characterization tests, the architecture presented in Fig. 2 has been used to assess the Characterizer's latency measurement accuracy.In particular, the results are summarized through the maximum value (indicated as "Max" in the following tables), the mean value (µ), and the standard deviation (σ ) of L WAMPAC values provided by the Characterizer.Tables I and II present the results of the tests performed using the NI cRIO-9068, RR of 50 and 100 frames/s, and 2 or 10 PMUs.
In particular, Table I reports the results obtained when the UDP protocol is set, while Table II refers to TCP protocol.It is possible to observe the remarkable stability obtained with RT hardware: up to an RR of 100 frames/s, a maximum latency of about 1 ms was measured in the UDP case and about 1.3 ms in the TCP case.It is important to highlight that the obtained latency values include not only the effect of the Characterizer but also the latency concerning the Emulator, thus guaranteeing that the found maximum values are actually upper bounds for the latency introduced by the Characterizer.Histograms of the WAMPAC latency measured for 15 min, considering 2, 5, and 10 PMUs, RR at 100 frames/s, and UDP small packet size.
It can be seen from the test results that µ remains almost constant as the number of emulated PMUs increases in the RT case.This, considering also that σ is practically constant too, means that the RT device, even if the number of operations to perform is higher, manages to keep its performance almost unchanged in the various tests until time constraints can be respected.This is also confirmed by Fig. 7, which reports the histograms of L WAMPAC measured when RR = 100 frames/s, small payload size, UDP protocol, and 2, 5, and 10 emulated PMUs are considered (red, green, and blue bars, respectively).From the figure, it can be seen that the histograms are mostly overlapping.
On the other hand, with an RR of 200 frames/s, the system exhibits a state of heightened strain due to an increased number of concurrent requests.The load level of the CPU shows peaks at 100% occupancy and, for this reason, the device has some slowdowns, leading to higher maxima of measured latency in some tests.These correspond to few outliers that Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

TABLE III
WAMPAC LATENCY RESULTS FOR UDP PROTOCOL AND GP HARDWARE do not significantly affect µ and σ but reduce Characterizer's capability to accurately monitor a given system.Higher RRs are not manageable with the considered RT platform.
Tables I and II also show that as the packet size increases, µ slightly increases, as expected.In addition, µ is generally lower using UDP protocol than using TCP.This is because UDP is a lighter and faster protocol than TCP.
Furthermore, the TCP packet usually has a larger header than UDP packet (in our case, the TCP header is 20 B while that of UDP packets is 8 B) considering the same number of synchrophasors transmitted.
Table III shows an extract of the results achieved with GP hardware with UDP protocol and the same test length (see Table I for comparison).For these tests, the Characterizer's software was run in a workstation equipped with two Intel Xeon Gold 6128 processors, 256-GB RAM, and Windows 10 OS.To synchronize the workstation, NetTime software [43] with a 15-min refresh interval was used.The results are much more variable and maximum values are quite pronounced due to frequent interrupts that can slow down code execution (the worst found result is about 90 ms for 7 PMUs, small packet, and RR = 200 frames/s).In this case, σ reflects the increased variability, whereas µ can be even lower in some tests than that obtained with RT hardware.Indeed, the used workstation provides more computing power than the used RT system but no determinism is guaranteed.As for TCP, all these considerations still hold but, as in the RT case, with generally higher µ values than UDP (the results are not reported here for brevity).
To clarify the differences in the results obtained with the RT device and the GP device, Fig. 8 shows the trends of the measured WAMPAC latency in case of two emulated PMUs over 15 min.From the figure, higher stability of the values obtained with RT hardware is evident, whereas latency measurements are characterized by the presence of outliers.Therefore, the characterization of a WAMPAC system using the RT system shows, as expected, stable performance over time.Nonetheless, the comparison reveals that the GP implementation, even if not highly accurate, can be considered a useful tool for preliminary compliance tests of WAMPAC functionalities and investigation of the system behavior.
To get further into Characterizer assessment, the various contributions to L WAMPAC are isolated and shown in Fig. 9, considering ten emulated PMUs, 50 frames/s, UDP protocol,   To verify the functionality in the presence of a network issue, an extra communication delay in a test over 15 min has also been added.The above test configuration has been used and the data stream of the tenth PMU is delayed by 1 ms starting from the middle of the test duration and leaving the other PMU streams unaffected.The test shows that PMU communication latency (and consequently also WAMPAC latency) has an instantaneous increment equal to the extra delay, as expected.The second set of characterization tests corresponds to longduration tests, which have been carried out using the RT hardware.The Characterizer's configuration is composed of two emulated PMUs, 50 frames/s, and medium packet size.To verify the stability and accuracy of the implemented system in the long run, 2-h tests are considered with both the UDP and TCP protocols.Fig. 10 reports the latency measured by the Characterizer for the UDP case and reveals that it remains stable even during long tests.Almost the same results as in Table I, with values always below 0.90 ms (1.03 ms for TCP), were achieved: in particular, µ = 0.66 ms and σ = 0.03 ms for the UDP case, while µ = 0.75 ms and σ = 0.03 ms for the TCP case.The TCP case can also be compared with [31], leading to an accuracy improvement of one order of magnitude, from about 10 to 1 ms.Furthermore, these results can be used to interpret the accuracy of the measurements performed in the experimental tests of Section IV-C.
Afterward, tests in the presence of network delay have been performed.A preliminary test was conducted to verify that the network emulator does not introduce significant extra delays (in addition to those set) into the measurement and control chain.For this reason, the network emulator has been configured to provide no extra delay.The test was carried out with both the UDP and TCP protocols, but for the sake of brevity, only the UDP case is shown in Fig. 11 (the results obtained with TCP are similar).In this test, µ = 0.70 ms and σ = 0.04 ms have been obtained.As expected, both the mean and standard deviation are slightly larger than in previous tests without the network emulator, but the additional latency is negligible (the network emulator introduces an average latency of approximately 0.04 ms).Finally, two additional characterization tests were carried out with the UDP and TCP protocols by setting a significant delay in the network emulator to emulate the communication delay between the WS and the WAMPAC actuator.Considering that the maximum delay of the round-trip time in a real test case reported in [31] was 36 ms, 18 ms has been chosen as a representative one-way network delay, thus emulating the latency of a real communication network.For the UDP case, µ = 18.70 ms and σ = 0.03 ms have been achieved.As expected, the obtained average value is shifted by 18 ms compared with the previous ones.
In all the short-and long-duration tests, no packet loss occurred, which is a crucial aspect to consider for using the instrument in the field.

C. Experimental Tests on a Real WAMPAC System
In this section, the experimental tests conducted on the real WS based on OpenPDC and described in Section IV-A are presented.First, the architecture shown in Fig. 6(a) has been used and 2-h tests have been carried out with both the UDP and TCP protocols.Fig. 12 reports the results for the UDP case and a substantially similar behavior has been obtained with TCP.A linear trend with a positive slope can be observed interleaved with higher latency bursts.The linear trend lasts about 100 min and yields a variation of about 20 ms.This behavior is thus representative of the OpenPDC operation.L WAMPAC goes up to 120 ms in the UDP case and 140 ms in the TCP case for a few instants, while progressively smaller delays follow such spikes.In these tests, an average of 9.47 ms and a standard deviation of 6.16 ms were obtained for the UDP case, while µ = 12.19 ms and σ = 5.76 ms were found in the TCP case.These results are intrinsically tied to the workstation and OpenPDC application configuration and characteristics.They also depend on the concurrent processes running on the workstation and can be partially improved by increasing the OpenPDC process priority.Several tests have been performed in this regard, highlighting the role of a non-RT system in the latency trends.However, an analysis of the application configuration and performance is beyond the scope of this article.
Moreover, two tests were carried out with the architecture in Fig. 6 Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.WAMPAC latency measured for 2 h through the connection to OpenPDC and network emulator with an additional delay of 18 ms, considering 2 PMUs, RR at 50 frames/s, and TCP medium packet size.
The latency values appear substantially shifted by the added delay with respect to the prior results, but the pattern is the same as in the previous test.
Finally, a further test was carried out using a commercial PMU and two PMUs emulated by the WAMPAC Characterizer connected to the OpenPDC, as shown in Fig. 6(c).The chosen commercial PMU uses an M-Class algorithm for synchrophasor estimation with RR = 50 frames/s and TCP packets of 92 B. Such PMU was characterized beforehand in terms of reporting latency, finding an average µ PMU = 132.43ms and a standard deviation of 0.08 ms, for about 30 min, since the standard [2] requires at least 20 min of observation.Therefore, to make the scenario more realistic, the same latency was configured for the PMUs emulated by the Characterizer.The test results (not reported here for the sake of brevity) confirm the conclusions of previous tests (e.g., that in Fig. 12) while highlighting the Characterizer's flexibility.Indeed, the average of L T is almost shifted by µ PMU with respect to L WAMPAC of the above test, as expected, and the standard deviation is almost the same.Latency peaks are more erratic and depend on the specific test configuration, but the influence of L PMU on the measurements is still evident.
The proposed measurement platform proved to be effective, with an accuracy well-suited for testing WAMPAC applications in real operating scenarios.This measurement instrument can then help TSOs during the prototyping and implementation phases, so that configuration and testing can rely on an accurately measured WAMPAC latency.This is a key factor, because, when WAMPAC latency is not assessed properly, automated control may not work and, in any case, cannot be optimized.The results also show that the GP version does not allow the accuracy required to verify real WAMPAC constraints, but it can help in preliminary tests.

V. CONCLUSION
This article has proposed a WAMPAC Characterizer that is flexible and adaptable to the different types of WAMPAC systems.Its accuracy allows testing different WAMPAC scenarios, with several PMUs and with various RRs, packet configurations, and communication network delays.While evaluating the WAMPAC system under test and its promptness, the proposed platform does not introduce any significant uncertainty-related decision risks.The proposed measurement platform can be used for both short-and long-duration tests also when WAMPAC systems rely on UTC synchronization for their critical tasks and is thus a promising tool for TSOs at design, prototyping, and preproduction stage.

Fig. 1 .
Fig. 1.Diagram of the considered architecture of a generic WAMPAC system, composed of PMUs, WAMPAC server (WS), and control actuator, with the associated latencies.

Fig. 5 .
Fig. 5.Picture of the used hardware and connections for laboratory characterization.

Fig. 6 .
Fig. 6.Schematic of (a) experimental test architecture, (b) integration of the network emulator, and (c) integration of a commercial PMU.

Fig. 11 .
Fig. 11.WAMPAC latency measured for 2 h through the integration of the network emulator without extra delay, considering 2 PMUs, RR at 50 frames/s, and UDP medium packet size.
(b), introducing an additional delay of 18 ms through the NetEm like in Section IV-B.The results are shown in Fig. 13 for the TCP case, leading to µ = 29.38 ms and σ = 5.33 ms.

Fig. 13 .
Fig. 13.WAMPAC latency measured for 2 h through the connection to OpenPDC and network emulator with an additional delay of 18 ms, considering 2 PMUs, RR at 50 frames/s, and TCP medium packet size.

TABLE I WAMPAC
LATENCY RESULTS FOR UDP PROTOCOL AND RT HARDWARE

TABLE IV LATENCY
COMPONENTS FOR 10 PMUS, 50 frames/s, MEDIUM PACKET SIZE, AND UDP PROTOCOL and medium packet size.L WAMPAC and L C (latency of the control packet) measurements, obtained with the Characterizer, are reported as a function of time together with L WE , measured internally by the Emulator, and the maximum L D,i obtained by the Emulator among the PMUs.TableIVshows µ and σ of the individual latency components measured during the test.These results indicate that the latency introduced by the WAMPAC Emulator can be safely considered to be lower than 0.1 ms.Obviously, this latency contribution is related to the complexity of the policy and the number of PMUs, but, thanks to the RT performance, it is still an order of magnitude below other latency contributions.