Digital Electrical Substation Communications Based on Deterministic Time-Sensitive Networking Over Ethernet

,

substations handling the operation of the power distribution subsystems, including their sensors and actuators. A fundamental aspect of this transformation is the communication technology amongst the different substation processes, which will be replaced by Ethernet networks (either on optical fiber or copper-based links), instead of using vendor-specific solutions.
These Ethernet networks have the potential of allowing a streamlined exchange of data and the establishment of redundant communication paths for critical flows but cannot enforce any delivery guarantees or ensure end-to-end determinism. Thus, in a substation scenario, this traffic is usually handled with a three-tier architecture supported by dedicated field buses. These tiers are the Field Level, which implements lower-level interfaces with sensors and actuators; the Bay Level, which includes the equipment controlling the operation of the substation; and the Substation Level, which defines communication interfaces with other elements from the power grid. Thus, the Field-level processes usually have to handle time-critical traffic, such as event alarms, trigger signals, or the interlocking mechanisms of closed control loops. This type of traffic is usually propagated using the IEC 61850 standard [4] protocol, which carries these messages from the Merging Units (MUs) of the substation to the Intelligent Electronic Devices (IEDs), where the corresponding monitoring, control, protection, and diagnostics tasks of the substation are implemented.
For its application in Smart Grid, the IEC 61850 protocol needs to be complemented with additional features; such as node discovery, reconfiguration, aggregation of different priority flows, or guaranteed determinism as bounded end-toend latency. Additionally, its messages have to be transmitted using Ethernet-based networks. There are several Ethernetbased approaches that can be applied for Smart Grid, such as PRP (Parallel Redundancy Protocol) or White Rabbit (WR) HSR (High-Availability Seamless Redundancy) [5]. The former allows the creation of redundant network topologies, whereas the latter uses a customized implementation of the White Rabbit synchronization stack to distribute timing information and to implement redundant ring topologies with ∼3 µs recovery times [3]. However, none of these systems can enforce any type of message delivery guarantee. Hence, TSN represents the ideal alternative to support the migration to Smart Grid, as it can implement a deterministic communication network with support for redundant ring topologies for select flows, as mandated by IEC 61850, with zero switchover time using its 802.1CB [6] component.
Consequently, this paper shows that TSN is a viable alternative for supporting substation communication flows in accordance with the specifications of IEC 61850 [7]. The operation of the system is characterized in-depth in the following sections, but further reliability studies can still be conducted following the guidelines proposed in [8]. Therefore, after introducing the motivations for the adoption of TSN, its main functionalities and components are described in Section II. Next, the architecture of the TSN network nodes used in this work is explained in Section III. Then, the feasibility of applying a TSN network for integrating Smart Grid communications is shown in Section IV with the deployment of an experimental Proof-of-Concept (PoC) setup in a reallife substation. The results of the initial PoC are further characterized with a laboratory test bench in Section V, where the effects of applying different configuration parameters can be studied. Lastly, Section VI concludes by showing the viability of applying TSN for Smart Grid. The effects of user-driven configuration on achievable determinism are also outlined, and future lines of work are presented, such as the development of a specific TSN profile for Smart Grid.

II. TIME-SENSITIVE NETWORKING (TSN)
This section is intended to give the reader a brief overview of the concept of Time-Sensitive Networking (TSN) and its application on Ethernet-based networks. Readers already familiar with the concept and operation of TSN technologies may directly proceed to Section III.
TSN emerges as a set of enhancements for Ethernet networks set forth by the IEEE Standardization Committees. Traditionally, Ethernet has been considered a robust technology that can fulfill the communication needs of generalpurpose applications and, as a result, its use became pervasive alongside the widespread deployment of the Internet. It is thus a well-known standard that has become the foundation of the so-called ''open world'' communications. These are the IP-based flows that make up the bulk of most of today's popular Internet applications; such as HTTP web browsing or audio/video streaming. Typical Internet traffic is Best-Effort (BE) in nature and is usually supported with Ethernet networks, which share the same service philosophy.
The scenarios where timing and determinism guarantees are required usually rely on specialized, vendor-specific solutions. This is the case of industrial plants, avionics systems, or sensor-actuator control loops in general. These implement ''closed world'' communications and typically make use of field buses, such as ModBus or CAN, to guarantee that their timing and real-time constraints can be met. In heterogeneous environments, their use can lead to complex architectures with different bus domains, hence making their integration and maintenance highly costly. Furthermore, some scenarios may even require a separate data network, which adds up to system complexity.
TSN allows the combination of the aforementioned Internet-like, ''open world'' communications, with the ''closed world'' traffic associated with industrial monitoring and control using the same standard-based Ethernet network. An initial approach to this was the development of AVB (Audio/Video Bridging) [9], which was geared towards professional multimedia environments. Later on, AVB was superseded and targeted to broader applications including industrial automation, hence giving rise to Time-Sensitive Networking. Its components are outlined next.
• System-wide Synchronization provides a common time reference for the network. This allows the nodes of a distributed TSN system to work synchronously and forward messages consistently with the time of the network. This reference is maintained by the gPTP service (802.1AS) [10], which implements a PTP profile tailored for TSN. In the context of Smart Grid, gPTP timing can provide synchronization on the order of tens of nanoseconds (experiment V-C1) for the substation equipment, far exceeding the ∼1 µs specification of IEC 61850 Part 5 [11]. As this reference will be distributed over Ethernet links, it is a safer alternative than the GPSbased systems that synchronize Phasor Measurement Units in some substations, adding protection against accidental or intentional GPS malfunction [12].  [14], [15]. Their action allows TSN systems to guarantee system-level determinism for critical messages and the coexistence with Best-Effort data in the same Ethernet network.
• System Configuration and Traffic Identification. TSN handles traffic according to its Traffic Class. Thus, it implements several mechanisms for specifying the network topology, the criteria for assigning messages to a given traffic class, or the GCL Schedule of the traffic shapers. This is accomplished with resource reservation protocols (802.1Qcc, 802.1BA) [16], [17], which disseminate these parameters throughout the system. The traffic classes identified with these parameters will be denoted with a VLAN-tagged TSN stream (802.1Q) [18], whose associated priority will indicate the forwarding queue of the TAS module that it will be assigned to.
• Reliability. Seamless Redundancy. TSN networks can make use of the 802.1CB [6] component for protecting highly critical messages by allowing their transmission over disjoint physical paths in the network using a standard-defined frame replication scheme. This feature is often required in scenarios where the delivery of timecritical messages has to be guaranteed. This is the case of Smart Grid, where the use of redundancy mechanisms is expected in IEC 61850.
This work shows the application of a TSN approach for Smart Grid using two purpose-built nodes, whose architecture is described in Section III.

III. IMPLEMENTATION OF A TSN-CAPABLE NETWORK NODE
The experiments presented in this work make use of two different TSN nodes: the WR-ZEN board and the MAIN TSN Switch. They are both based on the Zynq-7000 devices from Xilinx [19]. These devices are programmable Systems-on-Chip (SoCs), featuring a dual-core ARM Processing System (PS) for running an embedded OS (e.g., Linux, RTEMS [20]), and FPGA Programmable Logic (PL) for implementing HDL coprocessors that can interface with the PS. These two TSN nodes share the same underlying architecture but use different SoC devices depending on their role in the system.

A. THE TSN NETWORK NODES
• The WR-ZEN Node implements a two-port network device that can operate both as a TSN Listener (Receiver) or a TSN Talker (Transmitter) of highly critical messages. Hence, it implements a reduced TSN system in the relatively small, low-cost Z-7015 programmable SoC. Additionally, this node includes a dedicated HDL coprocessor (DIO) for digitizing analog substation triggers in order to forward them over the network as high priority TSN messages. This node repurposes the original architecture of the WR-ZEN board [21], which was originally used for distributing White Rabbit timing [22].
• The MAIN TSN Switch. This node implements a fourport TSN bridge that forwards different TSN messages amongst its ports in accordance with user-specified settings. The bridge functionality requires the use of a greater amount of FPGA resources to support a multiport implementation. This led to a design that uses the larger Z-7030 Xilinx device, which was integrated in the purpose-built MAIN circuit board. TSN Subsystem (red). The DIO coprocessor is an exclusive component of the WR-ZEN node and is highlighted with a blue box. The software components run on the ARM-based PS and control the operation of the aforementioned subsystems. These components include the RTEMS OS, the gPTP service, configuration APIs, or other user tasks.

B. TSN NODE ARCHITECTURE
• The Ethernet Networking Subsystem provides the underlying Ethernet service for establishing a functioning TSN network. It consists of several off-the-shelf components allowing the instantiation of ordinary Ethernet ports in the PL of the Zynq-7000 device. These components include DMA controllers, a lightweight 1G Ethernet MAC ported from an open core design [23], the Xilinx PCS/PMA core, and Ethernet transceivers (GTP blocks for the WR-ZEN or a dedicated PHY chip for the MAIN node).
• The Timing Distribution System (gPTP) provides the crucial timing synchronization required for any TSN system as specified in the 802.1AS subcomponent [10]. This component defines an implementation of PTP particularized for TSN, which was applied to Smart Grid using an FPGA-based design like that shown in [24]. Thus, the subsystem requires two main cores in the FPGA logic that will be coordinated by a system service of the PS implementing the gPTP protocol: The PTP Hardware Clock (PHC) and the Time-Stamping Units (TSUs). The PHC contains the internal time representation of the node, which is steered by the execution of the gPTP synchronization service. The TSUs will be used for retrieving time stamps associated with the exchange of gPTP protocol messages.
• The TSN Subsystem implements the essential elements allowing the establishment of deterministic communication flows. In Fig. 1, these elements, which are instantiated on a per port basis, are the TSN VLAN Core and the Time-Aware Traffic Shaper (TAS). The VLAN Core operates as an input and forwarding stage: it identifies different types of traffic according to user-defined criteria, which are then encapsulated into VLAN-tagged TSN streams and delivered to the appropriate port and queue over the AXI Switching Core. The Time-Aware Traffic Shaper works synchronously with the time reference supplied from the PHC. It forwards messages deterministically by periodically activating its priority queues according to a user-defined Gate Control List (GCL) Schedule.
• The Software Environment of the TSN nodes. The TSN nodes make use of Zynq-7000 devices with a dualcore Processing System that supports the execution of the embedded real-time RTEMS 5.0 OS [20] environment. This OS provides the framework to support the various software components, modules and services to handle the operation of the TSN system: a gPTP synchronization service, Ethernet network drivers, and configuration APIs. The synchronization service updates the time representation of the PHC with the gPTP protocol (802.1AS [10]) and was developed as a custom RTEMS port of the OpenAvnu Project [25]. The network drivers initialize the Ethernet Subsystem and were adapted into RTEMS from the Xilinx Ethernet Network drivers [26]. They were additionally customized to allow their interaction with the Timing Subsystem. Lastly, the system uses two different APIs. The gPTP API is used to pass configuration parameters to the gPTP synchronization service (e.g., oscillator quality). The TSN API is used to indicate the traffic classes and applicable GCL schedules to the TSN Subsystem.

C. CONSIDERATIONS ON FPGA FOOTPRINT
These experiments make use of two different flavors of TSN nodes: The WR-ZEN node and the MAIN TSN node. These are based on Zynq-7000 devices and have different resource requirements depending on their role in the network. Thus, while the former implements an End-System with just two Ethernet ports, the latter is a four-port bridge with advanced switching capabilities. The system architecture of Fig. 1 shows that some cores are instantiated on a per port basis, and that others have FPGA slice usage dependent on their number of bus interfaces, as is the case of the AXI Switching Core. Hence, resource consumption is bound to be dependent on the number of ports instantiated in a particular node. Consequently, the WR-ZEN node will require relatively reduced FPGA logic and can be fitted in the small, low-cost Z-7015 device (70% overall usage), whereas the MAIN node will require greater resources and will have to use the larger Z-7030 device (60% overall usage) to support its four-port design. These figures were achieved with a design that prioritized moderate FPGA footprint, which led to some compromises in the implementation, like the use of relatively modest 4 kB buffers for each queue in the traffic shapers to reduce the utilization of Block RAM primitives.

IV. EXPERIMENTAL VALIDATION. ELECTRICAL SUBSTATION FIELD TESTS
The main premise of this work is to show that a TSN system can manage to successfully integrate all the data flows of electrical substations, especially the critical ones, over Ethernet networks extending the capabilities of the communication buses currently described in the IEC 61850 standard. Specifically, the proposed TSN system can implement the underlying communication layer of a Smart-Grid-enabled substation and, as a key difference from existing approaches, it allows for the joint transmission over shared physical links of internal control signaling [27], monitoring messages, and digitized analog triggers associated with critical events, which typically required the use of dedicated analog interfaces. Consequently, a Proof-of-Concept system demonstrating the feasibility of the application of TSN for Smart Grid was deployed in the real-life substation facility of the local electricity provider Grupo Cuerva S.L. [28] in Granada (Spain). This facility features a typical substation environment for performing a number of field tests for the TSN PoC, and its components can be examined in Fig. 2.

A. THE SUBSTATION ENVIRONMENT
The substation architecture consists of a central control element, the CPX Substation Central Unit from ZIV Automation [29], that interfaces with the rest of the equipment in the facility. The CPX Substation Central Unit operates on the Substation Level and is therefore tasked with implementing a number of supervisory and control processes over the rest of the equipment operating on the lower Bay Level. Units such as the Transformer Control Unit (ZIV RTN) or the Line Protection Unit (7IRD) stand out amongst the elements in this latter level. The supervisory tasks over these units are supported using dedicated TX/RX optical fiber link pairs. These links do not implement a proper Ethernet network, but are rather used to efficiently implement an electrically isolated communication channel. Additionally, the entire substation can be remotely managed using a dedicated RF link that interfaces with the CPX unit.

B. THE SUBSTATION FIELD TESTS
The field tests conducted in this work consisted of the deployment of a TSN system operating on the Bay level of the substation facility in order to evaluate the performance of the application of a TSN network for the transmission of highly critical data, GOOSE control signaling, and Best-Effort messages. To this end, the experimental TSN demonstrator interconnected the 7IRD Line Protection Unit with other substation equipment, as shown in the diagram of the PoC system of Fig. 3 or the picture of the actual deployment in the substation facility of Fig. 4. These tests made use of two main switching devices (the MAIN TSN nodes), and two end-systems (the WR-ZEN nodes) for establishing the demonstrator TSN network. In this setup, one of the WR-ZEN nodes assumes a TSN Talker role (ZEN-Pub) and is tasked with digitizing the analog triggers produced at the 7IRD unit and forwarding them as critical TSN messages. These messages will be received at the other Listener end-system (ZEN-Sub), which regenerates the analog trigger on reception of these messages. The digitalization  and analog conversion of the critical messages is supported by the DIO feature of the WR-ZEN nodes. The network also uses two laboratory PCs for producing additional background traffic in the TSN system. These flows will be injected from the Emitter PC and forwarded to Receiver PC over the two MAIN nodes (MB0 and MB1). These flows include GOOSE substation signaling, which is generated with a custom application developed at CIRCE Foundation [30], and Best-Effort traffic, which is emitted with a general-purpose traffic generator [31].  • A high priority flow (VLAN Priority: 3/2). The messages resulting from the digitalization of the analog triggers originating from the 7IRD node which are forwarded from the ZEN-Pub to ZEN-Sub nodes.
• A medium priority GOOSE signaling stream (VLAN Priority: 1). The TSN stream between the two laboratory PCs simulating the presence of background substation signaling with the GOOSE protocol.
• A Best-Effort flow (VLAN Priority: 0). Bulk Ethernet messages simulating the presence of additional, Internet-like traffic, typically non-critical monitoring.

D. CONFIGURATION PARAMETERS FOR THE SUBSTATION FIELD TESTS
The substation field tests have the objective of comparing the performance of the legacy transmission mechanism of critical events in substation facilities to what is achievable with a TSN Proof-of-Concept system. To this end, two different experiments, whose settings are outlined in Table 1, were devised: Performance of the Legacy Analog System, for studying the former, and the TSN Proof of Concept, for the latter.

E. EXPERIMENTAL VALIDATION AT THE SUBSTATION FACILITY
This section presents the field tests carried out at the substation facility to demonstrate the feasibility of the application of TSN networks for supporting Smart Grid deployments. These tests translate into the implementation of two major experiments with the goals of characterizing the legacy systems of the substation and showing the potential application of a TSN network in this environment, respectively. An overview of these experiments is contained in Table 2, where each experiment is introduced, its goals are presented, its experimental setup and measurements are briefly described, and its corresponding outcome is outlined. The results of each experiment are also discussed in greater detail in the points IV-E1 and IV-E2.

1) PERFORMANCE OF THE LEGACY ANALOG SYSTEM
This experiment was meant to establish the reference, baseline performance of the legacy substation equipment that is used for propagating high priority signals and events. In the substation facility, this mechanism consisted of a dedicated analog link that transmitted critical events in the form of simple analog triggers. These triggers, which originate from the 7IRD Unit, are then delivered to a substation relay that interfaces with the appropriate controller.
Thus, the experiment characterized their propagation time along this analog circuit. As a result, it was found that the transmission latency was around ∼ 209 µs, as can be observed in Fig. 5a (Left). It can be seen that the dedicated analog link is a reliable means for transporting critical signals; however, this comes at the expense of forfeiting transmission speed, as the circuit incurs additional latency by activating an electromechanical substation relay, and larger deployment costs, as these links have to be set up separately for each type of event handled at the substation.

2) FEASIBILITY OF THE TSN PROOF OF CONCEPT
This experiment was used to demonstrate the feasibility of applying a TSN network to support the transmission of critical process data over shared Ethernet interfaces while other substation traffic is also present in the background, such as GOOSE or Best-Effort flows. This was shown with a Proofof-Concept setup that measured the propagation time of critical events that were digitized and forwarded over the TSN system. It was found that the same critical events originating from the 7IRD Unit could now be delivered within 30 µs, that is, around ∼ 169 µs faster delivery than the legacy analog system. This can be seen in Figs. 5b and 5c (Center and Right). This delivery time corresponds to the propagation time along the three hops of the network.
These results are a significant Proof of Concept of the application of TSN for Smart Grid-capable substations. It is also proof of its scalability and versatility, as multiple flows with different levels of criticality and priority can now be deployed sharing the same physical TSN link, thereby removing the need for costly dedicated analog interfaces with complex electromechanical components that were often required for propagating critical signals.
The level of performance of this TSN flow aggregation mechanism is largely determined by the configuration parameters supplied by the user, which can enormously impact variables such as the packet loss ratio or the end-to-end latency. The influence of these parameters could not be studied at the substation where the TNS PoC demonstrator was deployed, as the electromechanical elements of the 7IRD Unit and the substation relay can only be safely activated a limited number of times before causing excessive wear. Hence, this characterization was performed thoroughly in a controlled laboratory environment, where the TSN demonstrator was replicated and larger data sets could be compiled.

V. SYSTEM CHARACTERIZATION. LABORATORY EXPERIMENTS AND RESULTS
As introduced at the end of Section IV, different settings can vastly affect the outcome of the data processed and propagated in a TSN system. In light of this, an in-depth characterization of the system was carried out in a controlled laboratory setup that replicated the original environment of the substation. This was accomplished by characterizing the different elements of the system, namely the performance of the timing distribution mechanism or the end-to-end latency and packet-loss ratio attainable under different settings. As a result, it is expected that these experiments will provide some valuable insight into the production of meaningful configuration designs for TSN systems at Smart Grid substations.

A. ELEMENTS OF THE EXPERIMENTAL LABORATORY TEST BENCH
The laboratory test bench presented in this section is intended to replicate that of the electrical substation field tests so that the system can be characterized thoroughly. Thus, the setup used for conducting the laboratory experiments will be similar to that presented in Section IV, but will introduce a few modifications to suit the new laboratory testing environment. These are introduced below.
• The TSN network nodes. As before, the TSN network will be formed by two different types of devices: The End-Systems, which use the WR-ZEN nodes, and the main switching nodes, which are implemented with the two TSN MAIN nodes.
• The substation event simulator. As opposed to the field tests, which used the 7IRD unit for producing critical event analog triggers, the laboratory characterization made use of an off-the-shelf signal generator instead. This signal generator can produce analog triggers at varying rates, which will be used for producing a larger number of events for the system characterization than could otherwise be generated at the substation environment.
• The measurement instrumentation. The experiments characterize the performance of the system in terms of its timing distribution accuracy, propagation delay of critical messages, packet loss ratio, and attainable bandwidth. These two latter parameters will be measured using a regular network sniffing tool [32], while the first two ones will be measured with a TDC Counter instrument [33]. The TDC Counter will replace the oscilloscope given its efficiency to perform multiple back-toback measurements of end-to-end latency values, which are stored as large data sets that can be used to easily derive statistical indicators.
• The substation traffic simulators. The laboratory validation uses the same two laboratory PCs for emitting and receiving both Best-Effort traffic and GOOSE signaling. Traffic integrity statistics will be derived at the Receiver PC.
This experimental setup is only concerned with characterizing the performance of the TSN system and hence the analog transmission circuit found at the substation is not replicated for these experiments. A picture showing the laboratory setup that was built for conducting the experiments presented in this section is included in Fig. 6.

B. CONFIGURATION PARAMETERS FOR THE EXPERIMENTAL CHARACTERIZATION
The laboratory characterization has the goal of determining the effects that the application of different configuration parameters can pose on the achievable determinism of critical messages as well as the integrity of the lower priority flows traversing the network. To this end, five different experiments have been conducted to further analyze the operation of the elements of the TSN system, such as the traffic aggregation mechanism or its timing distribution component, by replicating the environment of the substation field tests in a controlled laboratory setup.
Each experiment will thus make use of different configuration sets aimed at producing diverse effects in specific aspects of the system; namely on its ability to guarantee timely deliveries of critical messages or the integrity of the lower priority messages. In general, TSN systems need to be supplied with two main sets of parameters for their operation: a set of Traffic Classes for associating different types of traffic with VLAN-tagged TSN streams and a traffic shaping policy (GCL). Traffic classes can be identified in the presented solution by providing specific Ethernet header fields, like the ''Destination MAC Address'', and an associated priority value (PCP code). In order to minimize resource usage, this implementation was customized to handle priorities ranging from 0 (Best-Effort) to 3 (critical). The traffic shaping policy will be defined by a main scheduling cycle time (T cyc ) divided into constituting intervals (I n ) and will have to be applied on the shapers of the egress ports of the nodes in the network. The combined action of the traffic classes and the scheduling policies are the main drivers of the achievable determinism, but the user also needs to provide routing information for each traffic class and configure at least one of the nodes in the system to operate as a gPTP synchronization Master. Table 3 contains a summary of the configuration parameters that were tapped for each subsystem of the TSN network in order to perform each characterization experiment.
In particular, the Routing settings can be examined in Table 4, which indicates that the critical traffic is exchanged between the ZEN nodes, whereas the GOOSE and Best-Effort flows are exchanged between the two laboratory PCs.
Furthermore, the scheduling policies applied throughout the experiments can be seen in the Tables for the Configuration Sets I (Table 5), II (Table 6), and III (Table 7). Sets I and II share the same structure, as they define a 4 ms periodic  schedule divided into three different intervals, with the main difference between the two of them being that Set II enforces a more restrictive policy that limits the transmission of critical frames to a designated slot with a segregated queue (Q2) from that of gPTP synchronization. Set III studies the effects on the variation of the end-to-end latency of critical messages and the integrity of the rest of the traffic in the network when a number of scheduling policies are iteratively applied to the system.

C. CHARACTERIZATION EXPERIMENTS
This section covers the laboratory characterization that expands on that performed at the substation facility of the TSN system. This is achieved through a series of experiments with a twofold goal. On the one hand, they will aim to characterize the attainable performance and operation of the system (Experiments I through IV), whereas, on the other hand, they will also attempt to delimit the effects that the application of different settings can exert on the achievable determinism of critical TSN flows (Experiment V).
A general overview of these experiments is presented in Table 8, where each experiment is introduced, along with its goals, the experimental setup that was applied, the measurements that were performed, and the corresponding outcome that resulted from the experiment. These outcomes are subsequently discussed in greater detail in the corresponding discussion sections for each experiment (Sections V-C1 through V-C5).

1) PERFORMANCE OF THE TIMING DISTRIBUTION SYSTEM
The timing distribution is one of the crucial components required to ensure proper operation of a Time-Sensitive Networking system. Thus, its task is to propagate an accurate time base that will be shared amongst all the nodes that are part of the TSN network to guarantee the deterministic forwarding of critical messages, as it enables the synchronous operation of the different time-aware queues throughout the network. Overall, the achievable determinism of the TSN system is limited by the robustness of its timing distribution service, that is, the gPTP component, which is characterized in this experiment.
This characterization was carried out by means of deriving the Allan Deviation (ADEV) [34], which was calculated by recording PPS time differences between the ZEN-Pub and ZEN-Sub nodes for 65 hours. The ADEV indicator was then derived and represented using the AllanTools toolset [35], as shown in Fig. 7. The results indicate that the system remains stable in the long term and behaves linearly as its phase noise corresponds to that of a Gaussian process. Furthermore, the plot shows that the timing distribution has a degree of accuracy in the vicinity of tens of nanoseconds (∼ 10 ns) for averaging times on the order of one second, which is in the same range as other commercial PTP-based solutions.

2) TSN FLOW AGGREGATION OVER ETHERNET LINKS. BASELINE SCENARIO
This test defines the baseline experimental case as it characterizes the system using a trivial configuration similar to that used during the substation field tests. The results can be examined in Tables 9 and 10, which show the transmission latency of critical TSN messages and the level of integrity of the lower priority flows in the network (GOOSE and Best-Effort), respectively.
The experiment measured the operation of the system for 300 seconds and found that the applied scheduling policy yielded minimized end-to-latency for the critical messages transmitted over the network. This minimized latency oscillates between 25.8 µs, which is associated with the message propagation time, and a peak of 38.7 µs, which is accounted for by the effect of interfering gPTP traffic sharing the same queue (Q3) as the critical traffic, as can be seen in Table 9. Hence, in this experiment the main variable affecting the determinism of the critical flow is its associated priority VOLUME 8, 2020 TABLE 7. Scheduling policy for the iterative sweep of different scheduling cycle times performed during Experiment V (V-C5). Each row corresponds to a specific iteration, which is associated with a given cycle time that is divided into three separate intervals. regardless of the interval distribution in the scheduling cycle, which in this case was set to a 4 ms cycle (T cyc ) with a 1 ms interval for the critical traffic (I 2 ). As the traffic shapers implement a strict priority selection mechanism when several queues are active during the same interval, having the critical flow share the higher priority queue of gPTP (priority 3), which must always be open, would yield this minimized endto-end latency.
Furthermore, it was shown that this particular policy allows the preservation of the integrity of the GOOSE substation signaling messages at the expense of the Best-Effort traffic, which undergoes minor degradation (< 1%) under this configuration (Table 10).

3) TSN FLOW AGGREGATION OVER ETHERNET LINKS. HIGHEST ATTAINABLE RATE FOR CRITICAL TRAFFIC
The goal of the experiment was to determine the highest generation rate of critical messages under the baseline configuration that allows their transmission without incurring any data losses. This was measured by totalizing the number of 93630 VOLUME 8, 2020 TABLE 10. Summary of the traffic integrity level for the medium priority and Best-Effort flows traversing the network in experiment V-C2, as indicated in the Packet Loss (PL) entries. In the table NTX and NRX indicate the number of packets sent/received, and BW(TX) and BW(RX) indicate the measured bandwidth on transmission/reception. The integrity of the medium priority GOOSE signaling is preserved at the expense of the Best-Effort messages, which undergo minor degradation (< 1%).

TABLE 11.
Transmission latency associated with the critical traffic from experiment V-C4. The test uses a segregated queue for the critical traffic (Q2) and a long scheduling cycle with a small service slot for Q2, resulting in significant end-to-end latency variation (∼3 ms). critical messages forwarded from the ZEN-Pub node to the ZEN-Sub node, and then detecting packet losses as non-zero counter totalization.
It was determined that the highest attainable critical message transmission rate was 670 Hz for this experiment and configuration set in particular, and that higher rates gave rise to congestion losses. Consequently, the subsequent experiments will generate critical messages at 100 Hz in order to produce larger data sets for deriving statistical indicators.

4) TSN FLOW AGGREGATION OVER ETHERNET LINKS. MODERATE LINK UTILIZATION WITH BEST-EFFORT TRAFFIC
This experiment studies the effects of the application of a more restrictive scheduling policy, whereby critical messages will now have a separate, designated queue (Q2) which only gets activated during a specific interval of the Configuration Set II ( Table 6). The synchronization queue (Q3) is always active. Thus, it is expected that critical messages will be prone to be severely impacted by the user-designed schedule applied in the experiment. This is shown in Table 11, where it can be seen that the end-to-end latency variation is directly related to the length of the scheduling cycle.
Specifically, this policy still allows the realization of a minimum ∼26 µs latency, which corresponds to the minimum propagation and processing time through the Ethernet links. However, the maximum end-to-end latency will now be determined by the relationship between the cycle time of the scheduling policy of the Configuration Set II and the length of time that the queue for critical traffic remains idle. Hence, it should be noted that unlike the case of Experiment V-C2, the choice of a priority value other than 3 for the critical messages will result in the fact that the scheduling cycle and its internal interval distribution will now be the decisive factors for establishing the end-to-end latency of a given flow. This is confirmed in the experimental data, where the ∼3 ms latency variation measured corresponds to the length of time that the critical message queue is inactive, as the applied policy defines a 4 ms cycle (T cyc ) with a 1 ms slot Values for the end-to-end latency associated with the critical flow obtained for each iteration defined in Table 7.
when Q2 is active (I 2 ). Besides, as the critical traffic generator is not synchronized to the TSN system time, the 900 µs standard deviation of the end-to-latency of the critical messages indicates that most critical frames are forwarded within the 1 ms slot when Q2 is active. This leads to the conclusion that a poorly designed schedule could result in catastrophic system operation, causing critical data to miss delivery deadlines or even experience congestion losses.

5) TSN FLOW AGGREGATION OVER ETHERNET LINKS. WORST-CASE END-TO-END LATENCY FOR CRITICAL TRAFFIC AND EFFECTS ON THE INTEGRITY OF LOWER PRIORITY FLOWS
The tests carried out in this section set out to delimit the influence of different scheduling policies on the attainable determinism for critical messages and the integrity of the lower priority flows in the network. This was achieved by applying the different policies defined in each iteration of the Configuration Set III ( Table 7). The results of the end-to-end latency for the critical messages under each iteration can be examined in Table 12, where it can be seen that the relationship determining the maximum latency variation between the schedule cycle time and the duration of the slot for critical traffic that was pointed out in V-C4 still holds, and can be described with the expression in (1).
This expression describes the effect that the configuration applied for the experiment has on the latency variation. Hence, it was found that the maximum end-to-end latency was determined by the duration of the scheduling cycle (T cyc ) and the length of the slot for critical traffic (I 2 ), with the additional contributing delays of the propagation time (27 µs) TABLE 13. Summary of the traffic integrity level for the medium priority and Best-Effort flows traversing the network throughout the iterations of experiment V-C5, as indicated in the Packet Loss (PL) entries. In the table, NTX and NRX indicate the number of packets sent/received, and BW(TX) and BW(RX) indicate the measured bandwidth on transmission/reception. GOOSE signaling is preserved, even though some of the policies applied cause significant degradation for Best-Effort traffic (> 20%).
through the Ethernet links of the TSN system (t prop ), and a peak processing time of 17 µs (t del ). This empirical derivation of (1) is meant to show that the user would be able to adjust the end-to-end determinism by supplying different values for T cyc and I 2 for the scenario under evaluation with the current choices of priority for the critical flow and network topology. For instance, a 384 µs cycle (T cyc ) combined with a 96 µs interval for the critical traffic (I 2 ) should yield a maximum latency of 332 µs, which closely resembles the experimental data for iteration 3 in Table 12 (MAX: 332.210 µs). Other scenarios with different network topologies or a greater number of flows and priorities might lead to different expressions that the user should evaluate and leverage to design configurations that can target the desired levels of determinism.
The level of integrity of the lower priority flows in the network for each iteration of the experiment was also examined in parallel, as shown in Table 13. These results have a twofold implication. On the one hand, the determinism of critical flows is a parameter that can be designed for to meet the requirements of a given system (e.g., delivery deadlines). On the other hand, there is a compromise with the integrity of the lower priority flows, which could experience congestion losses with growing cycle times and comparatively short service slots. This is the case of the iterations under study, where the Best-Effort can undergo significant degradation in some cases, while the GOOSE signaling remains protected given its lower bandwidth usage and higher priority.
It is important to note that the buffer depth (4 kB per queue) of the current implementation is an important factor in the integrity results. The Best-Effort traffic is generated at a constant rate of 50 Mbps using 1500 B frames and, since a general-purpose OS is used, occasional bursts may occur. As the Best-Effort queue can only hold two 1500 B frames at a time when it is idle, we have found that there are cycle/interval combinations where the occasional frame may be dropped (∼4 frames on average) for cycles shorter than 1536 µs, and others, like 768 µs, that manage to avoid dropping any messages altogether. GOOSE traffic uses a higher priority queue with frames of 168 B (26 messages per queue), but it could still be affected if the traffic were to be emitted in sufficiently large bursts. This was the case of the 192 µs cycle iteration. These effects would be mitigated using larger buffers.
Lastly, it is worth noting that the application of the scheduling policies of Set III produces the maximum end-to-end latency described in the expression in (1) when applied to the current system. However, more complex topologies with a greater number of flows may require the derivation of more complicated policies. These policies should be able to take into account the fact that TSN messages may have different associated forwarding times as a result of internal processing delays, impact of arbitration mechanisms, or the use of redundant paths (802.1CB). This latter case is especially sensitive, as the propagation times of redundant messages over different paths can be widely different. In the literature there are several works for calculating meaningful policies under such complex scenarios, and the work in [36] provides a useful framework for assessing the synchronization error in message forwarding associated with the application of a certain policy in systems with time-varying delays.

VI. CONCLUSIONS AND FUTURE WORK
This work has shown the feasible application of a TSN system to a substation environment to enhance the new Ethernetbased networks that are being deployed in these facilities during the transition to Smart Grid. It is expected that the use of Smart Grid will provide a unified framework for managing and handling all the different data flows of the substation. Ethernet-based networks already allow the transmission of the signaling data and non-critical messages that are usually propagated in these environments. In this context, critical event data messages, which cannot be handled by regular Ethernet networks, could benefit from the use of the proposed TSN system, which would allow their deterministic transmission alongside the other flows of the substation over the same Ethernet-based infrastructure. Hence, this would provide Smart Grid-enabled substations with a flexible networking stack allowing simultaneous propagation of ordinary IP flows, GOOSE signaling (IEC 61850), or the critical traffic typically found in the supervisory and control processes on the Bay Level equipment of the substation. This latter point was proven by devising and deploying a Proof-of-Concept TSN system in an actual electrical substation from a local power utility [28], where several field tests managed to show that a TSN system could be successfully applied to unifying all the communications in the substation over a shared Ethernet-based bus: critical messages carrying digitized trigger data, medium priority GOOSE signaling from the IEC 61850 standard, and Best-Effort flows.
Next, this work performed an in-depth characterization of the influence of different configuration parameters on the performance of the system in a controlled laboratory environment. Thus, this stage started with the evaluation of the gPTP synchronization, which was pegged to the tens of nanoseconds. After that, the ability of the TSN system to combine background substation traffic (GOOSE, Best-Effort) with critical flows was assessed, and it was verified that a deterministic delivery for the critical messages could be enforced. In this context, the influence of the scheduling policy on the end-to-end latency of the critical flows was determined to be the result of the combined effect of the application of different cycle times and interval durations in the traffic shapers: bounded latencies between 81 µs and 9.251 ms could be achieved with cycles (T cyc ) and critical intervals (I 2 ) between 48 µs and 12 ms. Hence, it was found that the design of the traffic-shaping schedule is the chief parameter governing the attainable determinism for a TSN flow, allowing the user to target application-specific requirements. Overall, this has shown that the system is highly versatile and scalable, given its multiple user configuration options for handling different types of traffic, and flexible enough to allow the deployment of distributed applications supported with highly accurate gPTP synchronization (tens of nanoseconds accuracy).
After this characterization work, we have planned a number of future actions, like the development of a user-directed utility for the centralized configuration of the entire system or the design of larger TSN switches. Further applications of TSN for Smart Grid domains could also be considered, like the definition of a Smart Grid Profile for TSN, its application for monitoring low-voltage grids [37], or the implementation of an OPC-UA interface over TSN for managing substation equipment.