Development of Power System Models for Distributed Real-Time Simulations

This paper proposes a general methodology for the development of power system models suitable for distributed real-time simulations (D-RTS) based on topology, simulator interfaces and data exchange. D-RTS have risen as functional alternatives that can combine remote and multi-vendor resources for large-scale power system simulations via a virtual connection. However, previous work focused on combination of separate models and not the performance of D-RTS when splitting a single monolithic model, failing to study the behaviors of D-RTS, including, synchronization and accuracy. The proposed methodology is used to develop distributed models of two widely used testbed large power systems, the IEEE Australian Benchmark model and the IEEE 300-Bus system. These testbeds are selected as they can be simulated as both monolithic and distributed models in available simulators in order to validate both the methodology and resulting model performance. The obtained results and comparison between monolithic and distributed models support the proposed approach and demonstrate the performance of D-RTS under both steady-state and transient operations in multiple scenarios.


I. INTRODUCTION
Modern power systems are evolving to more complex model configurations due to the rising demand for global energy consumption and the increasing penetration of a wide variety of new power system components and devices [1], [2]. Flexible ac transmission systems (FACTS), renewable energy sources (RES), battery energy storage systems (BESS), microgrids, and dc interconnections will become more commonplace in future power systems [3], [4], [5]. Therefore, decision making for planning, managing, and operating such power systems pose new challenges as they require extensive analysis of system operation scenarios within shorter time The associate editor coordinating the review of this manuscript and approving it for publication was Junjian Qi . frames [6], [7]. To ensure efficient, reliable, and resilient operation, these time frames can even occur in the domains of real-time or faster than real-time operation depending on the intermittency of energy resources and complexity of the system and its components.
Computer-based offline simulations have been widely used in power system studies. For example, power system modeling and analysis can be conducted in PSS/E, Matpower, DIgSILENT PowerFactory and PSCAD, among others, while GridLAB-D and OpenDSS can be used for the analysis of distribution grids. However, these computer-based tools cannot balance computational speed, accuracy, and fidelity for modern power system simulations [8]. New alternatives are required to better understand the current challenges faced by modern grids and reduce their computational time [9].
Real-time simulation (RTS) in electromagnetic transient (EMT) domain provides more capabilities to address the challenges of these modern and complex power systems [10], [11], [12] enabling: i) flexibility for accurate representation of power and energy systems, ii) detailed power system studies, iii) comprehensive analysis of power electronics based systems, iv) control and power hardware-in-loop (HiL) testing, and v) detailed performance validation.
Real-time EMT simulations of power systems, however, require significant computational capacity in order to solve mathematical equations at given small-and discrete-time step and keep pace with a real-world clock. Hence, the level of detail in large-scale power system models can be limited by the availability of RTS hardware. To alleviate this issue, realtime co-simulations and distributed real-time simulations (D-RTS) have been proposed in the literature [13]. On the one hand, co-simulations provide the capability to integrate different types of real-time simulator hardware to perform complex and large-scale simulations, combining unique capabilities of different simulators and utilizing computation capacity. For instance, a hybrid co-simulation is proposed in [14] to simulate electronics-based components in an EMT solver that is interfaced with the rest of the grid running in a phasor-based solver, combining the advantages of both modeling approaches. In [15], a co-simulation is implemented to enable the study of a coupled transmission-distribution network model. D-RTS, on the other hand, enables integrating multiple simulators to (co-)simulate a large model as subsystems in simulator hardware that may (or not) be located in different geographical sites. Compared with aforementioned co-simulation cases, interfacing real-time simulators with data communication and information sharing is a major challenge to realize co-simulation and D-RTS of large power systems, since subsystems in RTS are time-sensitive [16]. The challenges include inherent characteristics of data communication such as time-varying delay and packet loss [17], [18].
Addressing the associated challenges of communication interface has been the major focus of the existing literature which is a necessary step to provide a general procedure for reliable approach towards geographically D-RTS (GD-RTS) applications. Different frameworks have been adopted to integrate multiple real-time simulators and perform D-RTS. The types of real-time simulators utilized and their geographical locations define the overall configuration of the simulation hardware. Frameworks and methodologies are specific to, and depend on these configurations. The existing frameworks reported in literature adopt processing devices such as computer-based servers equipped with network interface cards for data exchange via network [17], [19]. These frameworks allow virtual connections between multi-vendor realtime simulators [20], [21].
Multi-rate RTS is an alternative approach that utilizes available simulator hardware resources. Multi-rate simulations interface subsystems with different simulation timesteps together. For example, RTDS has developed a multi-rate approach which allows subsystems (within the same system) to be simulated at multiple time-steps [22]. A ''main step'' is used for the main system while ''super-steps'' and ''substeps'', larger and smaller than the main step, respectively, are used to simulate different subsystems. Super-steps allow to simulate portions of the grid as a detailed and accurate equivalent while saving modeling capabilities. Sub-steps allow a more detailed modeling of high-frequency switching devices. Although different time-steps are used in the system, the model contains a concise level of detail that can better utilize simulation resources as different subsystems can be modeled with the required degree of accuracy.
The practical need for real-time distributed simulations depends on the specific requirements of an application and also the topology of the grid that needs to be simulated. Configuring large models for distributed subsystems is challenging and it is not always possible to achieve; such aspects of large power system simulations have not been addressed in the current literature. For instance, highly interconnected (i.e., meshed) grids increase the number of interfaces that need to be used as they require several splitting points. Moreover, D-RTS of meshed grids adds to the communication overhead and the amount of variables that need to be exchanged between simulators.
Existing work in the literature either combines independent models [13], [23], [24] or splits small and simple subsystems [18], [25] in order to demonstrate the increased capabilities of D-RTS. Nevertheless, there has not been a straightforward methodology for splitting models nor utilizing it in large-scale systems [26], failing to validate the proper implementation of D-RTS and the performance of the resulting model [27]. The value of D-RTS is enhanced by enabling large-scale models to be simulated at multiple hardware/locations. There is, therefore, a gap in verifying the performance of D-RTS against a single monolithic model as a reference, allowing assessment of accuracy and synchronization.
To bridge this gap, this article proposes a detailed development of distributed simulation models from benchmark models. By comparing the results between monolithic and distributed model directly, the benefits as well as the limitations of D-RTS are illustrated allowing for appropriate scaling towards even larger models. Specifically, the contributions of this work with the aim of validating the proposed D-RTS approach, are: i) a step-by-step methodology for model separation based on available resources, ii) critical considerations for selection of model splitting locations and their impact on results and use-cases, iii) defining the functions of a generic D-RTS framework and analyzing the impact of data exchange, delay compensation and subsystem synchronization in the results.
The rest of paper is organized as follows. Section II introduces the critical functions of D-RTS framework. A generic methodology for splitting large-scale power system models is proposed in Section III. Section IV presents a case study based on the IEEE Australian Benchmark model and defines the model configurations. Section V demonstrates the performance of D-RTS based on the case study as well as when splitting the IEEE 300-Bus Distribution system further validating the effectiveness and usability of the proposed methodology and framework. Finally, Section VI draws the main conclusions of this work.

II. DISTRIBUTED REAL-TIME SIMULATION (D-RTS)
The most common method to perform co-simulation is based on multiple physically connected simulators in a single location or laboratory. This method improves utilization of computation capacity for large-scale power and energy system simulations. RTS vendors, such as RTDS and OPAL-RT, offer modular and expandable solutions to perform large-scale simulations based on hardware located in the same site. However, the required cost and human resources can be a barrier to simulation resources and the scale of the simulation itself.
Distributed real-time simulation (D-RTS) is becoming an emerging solution, that allows (co-)simulation of large power systems using simulators located in (different) sites and exchanging data through a virtual connection. Furthermore, geographically D-RTS (GD-RTS) refers to the co-simulation of these large and complex systems enabled by multiple real-time simulators located in remote areas, combining facilities with unique characteristics, while overcoming demanding computational requirements [13]. GD-RTS provides the flexibility to perform large-scale power system studies and experiments allowing researchers to focus on their expertise and different aspects such as modeling, control, power electronics, and protection. Distributed RTS will pave the initial steps for the rapid development in energy research without geographic restrictions and utilization of simulator hardware resources.
There are, nevertheless, several key challenges associated with D-RTS, among which are: i) the lack of a universal framework, ii) network delays, iii) packet loss, iv) hardware restrictions (e.g., sending rate limitation, bandwidth of network, etc.), and v) lower fidelity of simulation data [28]. D-RTS does help increase the computing units for real-time EMT simulation, however, its use is not as straightforward as in the case of expanding a single simulator.
Convergence issues are critical for real-time EMT simulations. A model is developed for real-time digital simulation in order to be solved in a reasonable simulation time-step. Based on the equivalent circuit, the components can accurately be represented by differential equations, then integrated and transformed by trapezoidal integration rule. Real-time EMT simulation may meet convergence issues when simulating non-linear circuits [29]. This issue is more likely to occur in D-RTS, since the uncertainty of communication and signal conversion algorithm should be considered in the application of the method.
In addition, hybrid co-simulations allow combination of models in different types of solvers such as EMT and phasor-mode. As detailed model of two methods is different because of inherent differences in modeling and simulation approaches, the convergence point after fast transient should have a reconciling mechanism [30]. However, this convergence issue is less likely to occur in D-RTS, because subsystems are simulated in RTS with accurate results.

A. FUNCTIONS OF A D-RTS FRAMEWORK
A reliable framework for D-RTS is necessary in order to provide proper coordination of the distributed simulators and data management to perform an efficient and accurate simulation [32]. There are several functions associated with a D-RTS framework. Distributed models are configured as subsystems that run in different simulators. The framework interconnects and co-simulates the subsystems. Generally, the tasks of a framework includes: i) measurement of power system variables, ii) signal transformation, iii) data exchange, iv) signal reconstruction, v) time delay compensation, vi) synchronization between subsystems, and vii) signal injection.
A typical structure of a D-RTS framework is shown in Fig. 1. Currents and voltages are measured at the points where the system is split. The signals can be transformed by the discrete Fourier transform (DFT) or converted into the synchronous reference frame (dq0). By signal decomposition, three-phase measurements are sent as transformed signals instead of time domain signals, in real time, in order to obtain a higher fidelity. Simulators are equipped with communication interfaces allowing data exchange between simulators via a gateway server. The communication interface is a hardware module in the simulator and configurable by simulation software. A gateway server converts data in different protocols and manages them from and to varying sources and destinations, giving compatibility between multiple simulators. Then, the remote side reconstructs signals considering delay compensation and synchronization. Finally, reconstructed signals are injected into the distributed subsystem.

1) SIGNAL CONVERSION
Signal conversion includes transformation and reconstruction of power system signals. Instantaneous current and voltage waveforms are transformed to reduce the impacts from sampling and delays on simulation fidelity. Reconstructing signals allow to compensate delays, as phases can be shifted. Furthermore, the use of a remote phase reference allows to synchronize the simulation of subsystems. Therefore, the exchanged signals need to be transformed and reconstructed by a stable algorithm, which can represent all necessary information of the instantaneous current and voltage.

2) DATA EXCHANGE
A communication interface allows data exchange via network between subsystems. It is used to build communication between simulator and computer-based gateway server, which depends on the type of real-time simulator. The hardware to support external communications is typically embedded with the simulators. A gateway server tailored to D-RTS is necessary to manage and transfer data from different simulators. The requirements of such server and communication links are fast transmission speed, and sufficient bandwidth.

3) DELAY COMPENSATION
In D-RTS, the delay of one data packet to its destination includes the sampling, transformation, exchange and reconstruction time of such a packet. Delay compensation reduces or eliminates the impact of time delay on simulation performance because of the interconnection. Moreover, such function is important to generate comparable results with normal real-time simulation.
Other delays are related to the alignment of simulation results between local and remote subsystems. As these delays do not consume RTS resources, they do not have to be compensated during the simulation. By aligning simulation results from remote subsystems (relative time) with the local one (absolute time), these delays are accounted for and results of D-RTS are comparable with that of a monolithic model.

4) SUBSYSTEM SYNCHRONIZATION
Synchronization is critical in D-RTS as it confirms that signals are injected at the proper time with the correct phase. A commonly used method to synchronize subsystems remotely is based on GPS signals. GPS signals help to make sure that subsystems start their simulations at exactly the same time.
Alternatively, independent sources running as a phase reference in each subsystem can be used. The initialization reference is not affected by delays and starts with each distributed simulation. The phases for signal transformation and reconstruction in each subsystem are set by its local reference. Although the distributed simulations do not start at the same time, the local reference can synchronize with the phases for remote simulation regardless of time delay as initial phases can be shifted to compensate signal conversion delays.

5) INTERFACE ALGORITHMS (IA)
An interface algorithm (IA) is essential to interconnect distributed subsystems, which has influences on the stability and precision of D-RTS. The features of IA should have high accuracy, reliability, straightforward implementation and low time delay between subsystem for D-RTS. The IA allows to inject signals into subsystems, which is the last step to interconnect subsystems. For power HiL (PHiL), simulators co-simulate with hardware such as a grid emulator. At the power side, sensors and amplifiers are necessary to transfer suitable signals, due to the limited acceptable voltage on the real interface. Signals are exchanged between physical and digital parts. On the other hand, for simulator-to-simulator, a network and communication interface allow data exchange, and thus, the interface is simpler. Step 1 to consider when splitting a power system. Based on system complexity and the connection of series components (SCs), splitting points can be screened as easy, moderate or complex. An ''easy'' level refers to parts of the grid that have the simplest structure, usually interconnected to the rest of the system by few series components (e.g., single transmission lines). Moderate complexity locations usually include several series components (e.g., two double transmission lines linking three substations). Complex splitting points include highly meshed areas that interconnect several nodes in the system, and the split should generally be avoided at these locations. For example, a monolithic model can be split into two distributed models (DMs) if the splitting points are chosen to be in an easy or moderate area. There are one or two SCs linked at the splitting point in an easy area and more than two SCs connected at the splitting point in a moderate area. On the other hand, for a complex area, the split of a monolithic model creates more than two DMs and present several SCs, which means that more than one D-RTS framework is needed to split the system.

III. SPLITTING POWER SYSTEM MODELS FOR DISTRIBUTED REAL-TIME SIMULATIONS
Following identification of splitting points, the user can select areas most fit to split the system, reducing the complexity of implementing the D-RTS interfaces, and optimizing computational resource allocation. The most preferable points to split the system are those that present the lowest complexity (easy > moderate > complex), where fewer DMs are created and SCs need to be connected, as shown in Step 2 of the methodology.
After the selection of one or more splitting locations, Step 3 is to define the type and number of variables. For instance, the selection of harmonics depends on the system under study and the type of analysis. If a study focuses on analyzing harmonics, the exchanged number of harmonics to exchange should be increased.
Step 4 requires the selection of an interface based on available resources, and desired precision for the simulation. References [33] and [34] can be used to select the corresponding IA. Since the signals are injected by the IA in the subsystem, the selection of IA should provide accurate and precise data without causing numerical instabilities. Inclusion of the IA completes the distributed models and their allocation to the different simulators. As mentioned, conversion of signals is carried out by the IA, which is the main consumer of processing units in the D-RTS framework. Conversion methods define the execution time, thus, the computing load which can be used to predict the feasibility of one splitting point.
It is essential to confirm that all subsystems and the D-RTS framework can be simulated in each allocated simulator, which is carried out in Step 5. If there is a lack of computation resources, time-step increase can be considered, however, such increase will reduce the simulation accuracy and fidelity. Similarly, the user can adjust the number of harmonics that are exchanged between subsystems, if such an option is possible. In this case, the aim is to decrease the number of exchange variables and consequently the execution time for the conversion of signals. As another alternative, model rearrangement can be performed by selecting a different splitting point in the system and repeating steps two and three, until all necessary conditions are satisfied.
Once the subsystem interfaces have been established, the selection of communication interface is performed in Step 6, based on hardware availability and the requirements of the application. With the communication interface defined, the compensation time-step can be established. Here, initial selections can be made based on network measurements, however, the time-step can be further adjusted during initial validation simulations in order to obtain the smallest errors of D-RTS, at which point, the setup of D-RTS can be finalized.

B. KEY POINTS FOR SPLITTING A LARGE SYSTEM
Separation of a large model into smaller subsystems is a key step for distributed simulations as it has major impact on the simulation quality. In previous work, no systematic approach on where and how to split a system has been defined. There are multiple factors that should be considered when appropriately splitting a system, most critical among which are: i) the location of splitting points, ii) the number of series components, iii) the number of harmonics in a signal when using a frequency-domain framework, iv) the number of variables for communication, and v) utilization of subsystem interface. Splitting points define the locations where the system is split into subsystems based on the available simulation hardware, and are linked to the number of D-RTS frameworks to be included. The series components refer to any power system element which is arranged in a series connection between two areas (e.g., transmission lines and transformers). The D-RTS framework is added at the end of series components allowing the split of the system. The number and type of series components are important for the computational resources used by the subsystem interface as it relates to the data and communication overhead introduced for D-RTS at each splitting point.
As frequency-domain based frameworks offer better performance, the number of harmonics that are used in the conversion to the frequency domain defines the harmonic content of the different variables (e.g., voltage, current, etc.) that need to be transferred via the network. As mentioned in Section II-A1, signal transformation and reconstruction are both based on a Fourier analysis or synchronous reference frame (dq0). A larger number of harmonics will include higher level of detail for the system and thus, it will improve simulation fidelity. However, the computational resources of a single simulator and the size of exchanged packets are limited, restricting the number of harmonics to exchange between subsystems. The number of exchanged signals can be an indicator for computing load of D-RTS framework, and can be calculated as: where N V , N SC , N P , N T , N H , and N C are the number of variables, series components connected to the splitting point, phases in ac links or poles in dc links, transformed signals (e.g., DFT, dq0), harmonics (considering fundamental frequency), and control signals, respectively. One signal is added to N H to account for the dc offset. Additional signals that can be used to control a remote subsystem (e.g., trigger signals to update plots, trip signals for breakers, etc.) are included in N C . For instance, Table 1 shows the execution time of DFT and dq0 algorithms for the fundamental frequency in PB5-based RTDS simulators. This execution time increases linearly with the growing number of harmonics and signals to be converted.

C. SUITABLE CASES FOR D-RTS
Although D-RTS can provide accurate results when properly designed, not all power system models and simulation cases can be considered well-suited due to the limitations imposed by D-RTS simulations. Model separation should consider any suitable splitting points and the availability of sufficient resources for each subsystem to successfully perform the simulation tasks. The splitting areas depend on the specifics of the analysis/study. If an appropriate area cannot be chosen based on this criteria, the topology of the model starts to play a major role.
Meshed systems tend to be more difficult to split while radial and longitudinal power systems (LPS) are less complex in that perspective, due to the smaller number of interconnections. The proposed methodology is efficient to evaluate the separation complexity of meshed power system models to lower the difficulties for selecting suitable splitting point and accommodate distributed models with available simulation resources. As a result, areas interconnected by few tie lines are preferable locations for separation of a larger system. The number of exchanged variables is also reduced which allows to maintain higher simulation quality, while reducing the required D-RTS resources. Regardless of model complexity, if separation leads to few subsystem interfaces, the task of time delay compensation is also simplified, i.e., systems with highly meshed subsystems but with several proper splitting points with few series components are still suitable cases for D-RTS. In contrast, small but highly meshed systems lead to highly complex D-RTS scenarios. Taking advantage of model topology creates certain splitting points where models can be split, e.g., high voltage, direct current (HVDC) links provide natural splitting points as they split ac systems. For instance, a strong case for D-RTS can be made for power systems with multiple HVDC links such as the China Southern Power Grid (CSG), the National Energy Grid of Brazil and the National Electric Grid in India, or for LPS such as the power systems of Australia, Chile, Mexico, New Zealand, and Peru.

IV. IMPLEMENTATION OF DISTRIBUTED REAL-TIME SIMULATION
As analyzed above, LPS present suitable use cases for D-RTS. These models present a radial configuration (with multiple infeeds), with generation areas electrically distant from load centers, connected through long transmission lines [35]. For instance, the Australian National Electricity Market (NEM) is one of the longest power systems that spans Australia's eastern and south-eastern coasts covering a distance over 5,000 km [36]. The Chilean National Electricity System (SEN) extends for more than 3,100 km [37]. The major load is Santiago, Chile's capital, located in the middle of the system which is supplied by hydroelectric power plants from the south and thermal generation and solar PV from the north. New Zealand's National Grid expands for near 2,300 km and its primary source of energy is hydroelectricity [38]. Most of the power is generated in the South Island but it is mainly used in the North Island. As these systems interconnect areas over long distances by few tie lines, the separation complexity is low. As a result, D-RTS provides an alternative to simulate and study LPS without losing accuracy.

A. DESCRIPTION
The first power system used in this work is the IEEE Australian Benchmark model [39]. This testbed grid has been mainly used for analysis and control of small-signal stability phenomena [40], design of controllers [41] and integration of HVDC systems [42] in phasor-based software. Recently, the model has been converted to allow EMT real-time simulations [43], [44] and expanded to include the integration of advanced energy conversion systems and high-switching frequency power electronics converters [45].
The model is a longitudinal multi-machine system loosely based on the Australian NEM. It contains 5 strong areas interconnected by few tie lines, 14 generators with their corresponding exciter, stabilizer and governor, 5 static VAr compensators (SVCs), 59 buses, and 104 lines [39]. Furthermore, the model spans for 3,300 km and more than 66% of transmission lines have a length over 100 km [43]. The one-line diagram of the 50-Hz system, with splitting areas identified, is shown in Fig. 3.
Remark: The topology and characteristics of the IEEE Australian Benchmark model makes the system an excellent demonstration for the proposed methodology and the practical application of D-RTS. This model is also chosen as it can be simulated simultaneously as a monolithic and distributed system in the available simulation facilities in the laboratory, allowing direct comparison and validation of results. Fig. 3 identifies sections of model separation, as discussed in Section III. For this system, three different categories, based on complexity, are identified. While strong areas (light red in the figure) are difficult to split due to their mesh configuration, tie lines (light blue and green in the figure) are natural points to split the system. In particular, tie lines which present a reduced number of circuits are preferred (light green).

B. MODEL CONFIGURATIONS
Three different configurations of the IEEE Australian Benchmark model have been derived. These are: • Monolithic model (M): The original model that will be used as reference to assess and validate the performance of D-RTS for a large-scale power system. The model is simulated in 3 PB5-based RTDS racks as shown in Fig. 4(a) [43]. • Distributed model 1 (D1): Area 5 is simulated in a remote rack as illustrated in Fig. 4(b). The model is split at Bus 509 of line 315-509. Area 5 represents Subsystem 1 and the rest of system Subsystem 2.
• Distributed model 2 (D2): Area 4 is simulated in a remote rack as shown in Fig. 4(c). In this case, the model is split at Bus 205 of line 205-416. Area 4 represents Subsystem 1 and the rest of system Subsystem 2.
The D-RTS framework, implemented at one end of the tie line of the corresponding bus, is used to exchange data between distributed subsystems, via the Internet. The ITM algorithm is chosen as the IA [33], [34]. ITM uses controlled voltage/current sources to directly impose voltage and current at interfaces on both sides, respectively. The selection of Thévenin/Norton equivalents consider impedances, and time delay at both sides of the interface. Controlled current (voltage) sources are utilized to represent a weak (strong) subsystem with a large (small) equivalent impedance in a stronger (weaker) subsystem with a small (large) equivalent impedance.
In both distributed models, Subsystem 2 includes four out of five areas, while Subsystem 1 represents the remaining area of the monolithic model. Since Subsystem 2 is a large area including more components/branches connected in parallel, its equivalent impedance is smaller than the one in Subsystem 1. Moreover, the monolithic model is split at the end of long transmission lines that have large impedance by nature. The whole transmission lines are included as the interface of Subsystem 1, leading to have a large terminal impedance. Therefore, Subsystem 1 is represented by controlled current sources (in Subsystem 2), when tailoring the ITM interface [31]. Subsystem 2 has a relatively small equivalent impedance, and thus, is represented as controlled voltage sources (in Subsystem 1). If system impedance cannot be evaluated in a simple manner, the methods proposed in [46], [47], [48] can be used. In practice, remote areas (i.e., Areas 5 and 4 for D1 and D2, respectively) are modeled based on controlled current sources, while the interface for the rest of the system uses controlled voltage sources ( Fig. 1 shows this implementation).

V. RESULTS AND VERIFICATION
PB5-based RTDS simulators are used to simulate the models with a time-step of 50 µs. The D-RTS framework uses a total of 25 signals including: dc offset, fundamental frequency, second and third harmonics of each signal, and a control signal to trigger the update of the plots 1 (refer to Eq. (1)). These signals are exchanged through the internet with a rate of 500 packages per second (10 times of fundamental frequency, 50 Hz). Results include steady-state analysis and transient behavior during different disturbances. In order to compare the performance of D-RTS, all errors are calculated based on the monolithic model.

A. DELAYS
Time delays exist in network-based virtual connections for D-RTS. Such delays can be measured between simulators, and include sampling and transmission delays but exclude delays from signal conversion. Based on the connection arrangement of the simulators, the one-way transmission delay includes twice the delay between processor and network cards plus the delay from sending data from the local network card, through the gateway, to the remote network card. To measure the round trip time for data exchange in the laboratory setup, each simulator is set to send 500 packets per second for a period of 400 s. The average round trip time for data exchange is shown in Fig. 5 and its average value is 2.8 ms, thus, oneway delay is approximately 1.4 ms.

B. STEADY-STATE SIMULATION
Active (P) and reactive (Q) power errors are obtained for all generators in the model, and voltage errors are calculated at the high voltage side of each generator. Using the monolithic From Table 2, the maximum | P| is 0.44% and the maximum | Q| is 0.61% for D1 and D2, respectively. The errors measured in power for D-RTS are below 1% compared to the monolithic model, demonstrating acceptable accuracy for the D-RTS. Differences in voltages and phase angles are shown in Table 3. The largest error of bus voltage is -0.48% on Bus 410 in D2; the error of Bus 505 has a difference of -2.22 • , which is the largest error in phase angle measured in D1. Based on the steady-state results, the distributed models have similar behavior with the monolithic model which confirm the validity of steady-state analysis in D-RTS.

C. TRANSIENT ANALYSIS
The synchronization mechanism between the subsystems limits the impact of network latency on steady-state analysis. However, in transient analysis, communication delays affect all signals transferred through the D-RTS framework. Such 2 Angle differences are given in degrees. The transient behavior of distributed models is illustrated and compared for four distinct cases: i) a single line-toground fault at Bus 508; ii) a two-phase to ground fault at Bus 407, iii) the sudden disconnection of generator TG 5, and iv) under system-wide oscillations. Figures 6 and 7 present the results for single line-to-ground and two-phase to ground faults, respectively. Voltages at the faulted bus, currents at one transmission line connected to that bus and currents at one representative line of the splitting area are shown. Moreover, the errors in phase A for voltages and currents are obtained with the monolithic model M as a reference.
The location of the D-RTS framework is expected to influence the simulation results. By comparing the results of the two distributed models attention is given to the performance of both the local and the remote model in each simulation.
The errors, in percentage, of each simulation time step between the monolithic and a distributed model are obtained by (3).
where x M (t) and x D (t) refer to the instantaneous values of the monolithic and distributed model, respectively, andx M represents the peak value of monolithic model in steady state.

1) CASE 1-LINE-TO-GROUND FAULT ON BUS 508
Fig . 6 shows the results of the three models for a single line to ground fault on Bus 508, which occurs at t = 0.1 s with a duration of 0.1 s. In Fig. 6(a), the voltage of Bus 508 is shown for all three models. Compared with the monolithic model, voltages are similar; both distributed models agree with the reference model. This is also verified by the two current waveforms shown for different areas of the power system. As the faulted Bus 508 is one of the closest buses to the D-RTS framework in D1, there are some small errors in steady state due to the interconnection framework. However, the difference is small and considered acceptable for the purposes of D-RTS. In contrast, the splitting location is electrically far away from the fault location in D2. Thus, based on results of Fig. 6, there is no significant error in the corresponding fault response for D2.
In any case, all differences between the results of the monolithic and distributed simulations are within acceptable tolerances; the maximum error of voltage in phase A is 5.97 kV for D1 and 0.0164 kV for D2. Thus, their largest errors are 2.55% for D1 and 0.7% for D2 related to the voltage of Bus 508 in phase A. The errors regarding current for D1 are also greater than that of D2. The maximum instantaneous error of the current between Buses 505 and 508 during the transient response are 5.1% and 0.022% for D1 and D2, respectively. For the remote area, the greatest errors of current between Buses 312 and 313 are 5.65% and 0.0077% for D1 and D2, respectively.
Although there are small oscillations in the voltage of Bus 508 in Case 1, the overall performance of D-RTS is considered acceptable for both distributed models. Specifically for D1, since Bus 508 is one of the closest buses of D-RTS framework, the results are more affected by the limitations of the D-RTS framework than other remote buses. The causes of the differences can be attributed to the limited number of harmonics, possible package losses and time delays. In Fig. 6(a), the magnitude of oscillation in D1 is close to 0.8% in steady state (after 0.25 s).

2) CASE 2-TWO-PHASE TO GROUND FAULT ON BUS 407
The results for a two-phase to ground fault on Bus 407 is shown in Fig. 7. Similarly to the observations of Case 1, the differences measured for both D1 and D2 are small and acceptable for the purposed of D-RTS. Here, the differences in model D2 are relatively larger than those measured in D1 for the transmission line close to the location where the model has been split. The maximum voltage error of Bus 407 is 0.005 kV (0.002%) in D1 and for D2, it is 4.38 kV (1.82%). The currents between Buses 406 and 407 have a maximum error of 0.0033% in D1 and 1.66% in D2. For the splitting area regarding the current between Buses 211 and 214, the largest deviations are 0.24% for D1 and 8.16% for D2, all of which are within acceptable tolerances.
In general, Case 2 has a similar performance to Case 1 validating the overall proposition for D-RTS. Specifically considering the location of the D-RTS framework in the two distributed models, Bus 407 in D2 is further away from the splitting point than Bus 508 in D1. This explains why the differences in the measured voltage and current on Bus 508 in Case 1 are greater than what is measured in Bus 407 in Case 2.

3) CASE 3-GENERATOR TG 5 TRIP
The two previous cases have shown a detailed comparison for specific voltages and currents for one bus or transmission line, respectively. In Case 3, P and Q of one generator per area are used to represent the performance of the models when the largest generator (TG 5) is suddenly disconnected from the system.
Results are shown in Fig. 8 with the response of the same generator plotted in one single subfigure, allowing comparison of its behavior across the three models, when TG 5 trips. In all three models, the response of the generators is similar, VOLUME 10, 2022  which means that distributed models give acceptable results related to power of generator. Case 3 helps to further verify the performance of D-RTS.

4) CASE 4-SYSTEM-WIDE OSCILLATIONS
The behavior of the distributed models is also compared against the monolithic one under a large-signal instability event. Here a substantial change in the active power reference of TG 2 is applied which leads to severe system-wide oscillations, as the system cannot reach a new equilibrium point. The output power P of TG 2 and the power flow of the interconnectors between the different areas of the system are compared for all three models in Fig. 9. The change in the setpoint takes place at t = 1 s, and system oscillations are observed from t = 7 s onwards. Under these conditions, the two distributed models (D1 and D2) behave in a similar way to the monolithic model M, demonstrating the validity of well-designed D-RTS models for system-wide oscillation events. It should, however, again be noted that D-RTS cannot replicate all possible faults in the system and in these cases, alternative means of simulation will be required.

5) CASE 5-IEEE 300-BUS DISTRIBUTION MODEL
In order to further validate the proposed methodology, the use of D-RTS is also demonstrated on a larger and more complex benchmark power system model, the IEEE 300-Bus Distribution system. This testbed model is a 60-Hz system which contains 69 generators, 304 transmission lines and 195 loads [49]. Due to its highly meshed structure, most of the potential splitting points are screened as complex or moderate (as defined in Section III-A), requiring several interfaces and computation resources. The most suitable splitting point is at Bus 37 of the transmission line 37-9001, as shown in Fig. 10. The monolithic model (M 300 ) is simulated in 6 PB5-based RTDS racks, while the distributed model (D 300 ) requires 5 racks for the local subsystem and 1 rack for the remote subsystem.
In order to demonstrate the implementation of the D-RTS framework, a three-phase-to-ground fault is applied at Bus 48 at t = 0.1 s which is cleared after 100 ms by opening the corresponding circuit breakers. Results, including voltage at the faulted bus and currents of two neighboring transmission lines, are shown in Fig. 11. The maximum voltage error of Bus 48 is 2.98%, which occurs for a short period of time when the fault is cleared by opening the breakers. Similarly, the largest errors of transmission line currents are 6.31% and −9.74% for line 48 to 107 and line 99 to 107, respectively, when clearing the fault. Even though the model is larger and more complex than the IEEE Australian Benchmark model, all errors are within an acceptable tolerance for this critical transient operation of the system in a case where the split can be achieved.

D. DISCUSSION
Although there is approximately 1.4 ms one-way communication delay between subsystems, simulation results show that D-RTS are numerically stable. Even in transient simulations, no numerical instabilities and time misalignment are observed in all five cases. To generate these comparable results, synchronization and delay compensation issues in D-RTS need to be addressed. Independent sources running   in each distributed subsystem generate their local phase references for the alignment of simulations. Time delays (certain number of time steps), due to signal conversion (e.g., DFT-based method) are compensated by utilizing a phase shift, when the received quantities are reconstructed back as time-domain signals. Appropriate time delay compensation is achieved when the power flow (P and Q) at splitting points is similar for both sides [16]. Finally, communication delays lead to delayed data acquisition, which is eliminated by aligning the relative time of the remote side with the absolute time at the local side. This can also be achieved with time synchronization (e.g., GPS). The simulation results are acquired in real time, while the alignment of captured datapoints is a post-processing step to obtain comparable results for different simulations and models.
Based on the results of Cases 1 and 2, it is shown that the location of the D-RTS framework may impact D-RTS and corresponding analysis but the differences observed are within acceptable limits. This also indicates that the split of the system is preferable done further away from the areas of interest in distributed models; a conclusion helpful to understand the impacts of D-RTS framework on related simulation results.
Overall, as D-RTS frameworks have been added to split large power systems, results of distributed models are not expected to be exactly the same with the monolithic model. However, based on the comparison of errors, both D1 and D2 of the IEEE Australian Benchmark model have an acceptable performance. Moreover, the implementation and performance of D-RTS is validated on a larger model, the IEEE 300-Bus system. Results demonstrate that errors are within acceptable margin monolithic (M 300 ) and distributed (D 300 ) models near the splitting point location. All cases have demonstrated close performance between distributed and monolithic models, which demonstrate the steady-state and transient performance of D-RTS for large-scale power systems and the proposed methodology for model separation.
The benefits and limitations of D-RTS in comparison to other co-simulation setups have been summarized in Table 4. D-RTS fully takes advantage of real-time simulators, which can provide the highest accuracy (even in fast transient scenarios) and fidelity for large-scale models. Collaborative partnerships are required to employ D-RTS globally, and common D-RTS framework is used for remote coordination via the public Internet. In addition, security and confidentiality of models and data can be handled in D-RTS, since interfaces only require real-time data exchange at the splitting points.

VI. CONCLUSION
A methodology to split power system models for distributed real-time simulations (D-RTS) has been proposed in this paper, based on the characteristics of the power system and D-RTS framework selection. In addition, a reliable framework is shown to be critical for D-RTS, due to the requirements on data fidelity and network transmission for real-time simulations. The major difference to prior work in the context of D-RTS is the direct comparison against a single monolithic model. This paper analyzes the detailed behavior of D-RTS by splitting benchmark models of suitable size, showing its proper feasibility. This paper demonstrates the value of D-RTS for splitting large-scale power systems for real-time EMT simulations, as an alternative solution.
The methodology is implemented and demonstrated using the IEEE Australian Benchmark and IEEE 300-Bus systems. Results of the benchmark models show that appropriate performance and error between monolithic and distributed models are within acceptable limits for both steady-state and transient responses. It is also shown that the location of D-RTS framework, network delays and the amount of exchanged data, including the number of harmonics in the signals, influence the D-RTS within an acceptable range. D-RTS approaches, as a result, could be generalized for suitable power systems in order to increase the computation capacity by combining remote simulation resources and further supporting the case for laboratory interconnections. Finally, future work could be to validate more specific applications of different power system models for D-RTS.