Co-Simulation of Transmission and Distribution Systems—From Modeling to Software Development

As the penetration of Distributed Energy Resources (DERs) increases, it becomes critical to model these resources adequately in bulk power system (BPS) operation and planning studies. Adequate in this context refers to capturing the characteristics of the distribution system to the required degree such that reasonable estimates about BPS stability can be ascertained. Historically, performing such studies on a large transmission system connected to multiple distribution systems have been limited due to the unavailability of software capable of executing high fidelity simulation at the required scale on personal workstations i.e. not having to resort to high performance computing resources. In this paper, we present TDcoSim, an open source software that allows user to perform large scale high fidelity simulation for BPS planning and operation studies and (a) outline the necessary details required to build such software, (b) present the practical challenges faced during the development of TDcoSim.


I. INTRODUCTION
The impact of increasing penetration of distributed energy resources (DERs) on the operation and reliability of the bulk power system (BPS) is becoming more evident. This has piqued the interest of regulators. For instance, the North American Electric Reliability Corporation (NERC) created a working group to address system planning impacts from distributed energy resources [1]. The increasing presence of DERs in the distribution system has required regulators, planners, and system operators to reimagine and rewrite existing practices for system operation, planning, and standards. As such, system planners and operators have realized the need for advanced tools that can adequately capture the impact of The associate editor coordinating the review of this manuscript and approving it for publication was Min Wang . these intermittent and variable DERs on BPS stability in simulation models. The existing software tools used by system operators may need to be modified or upgraded for analysis of transmission and distribution system that has become increasingly coupled with high DER penetration levels.
For the power grid of the future, theoretical arguments can be made in favor of coupled transmission and distribution analysis, referred from here on as T&D analysis. While looking at BPS stability from the viewpoint of the system operator it should be noted that the primary intent is to have better estimates of the post-disturbance equilibrium conditions on the transmission(T)-system, which depends in part on distribution(D)-system. The equilibrium on the D-system, among other things, depends on DERs. Hence, the natural questions one tries to answer are with regards to DER ride-through behavior, recovery following VOLUME 10, 2022 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ disturbances, etc. Answers to these questions are fundamentally tied to local voltages, DER interconnection standards and settings/parameters, configuration of the T&D interface, etc. A T&D modeling approach using detailed models at the D-system naturally allows computation of local voltage at DER terminals, which is especially important in long distribution feeders where the voltage drop across the feeder can be greater than 5%. Furthermore, the T&D approach allows modeling D-systems with mixed vintages (different interconnection standards) of DERs to the best possible extent as DERs are represented individually. From a practical viewpoint, transmission planner is also interested in quantifying the impact of local voltages, DER interconnection standards, and DER settings/parameters on BPS stability assessment. With this information, planning engineers can pick and choose when T&D models are needed and when T-system models with aggregated representation of D-system loads and DERs would suffice. Ironically though, making such a decision still requires comparison of T-system only approach to T&D approach. Hence, software capable of T&D analysis is needed.

A. CURRENT BEST PRACTICES & CHALLENGES
With the rise in DER penetration levels, planning engineers are looking for ways to accurately understand the impact of DERs on grid reliability. The current best practice taken by transmission planners is to use positive sequence based aggregated models to represent the entire D-system and its associated loads and DERs at appropriate T-system load buses. As this approach uses aggregated models for loads and DERs, care should be taken to parameterize the models appropriately at different operating conditions to yield fair representation at the respective T-system-load buses. While the aggregated modeling approach provides a computationally efficient way to investigate the impact of DERs on the BPS, they do not provide additional insights on the impact of DERs on the D-system itself, and possible stability issues with the DERs themselves. The aggregated modeling approach also leaves out protection components within the D-system and also does not account for the impact of operation of D-system components such as regulators, switched capacitors etc. on DER operation.
While easier to implement, the current state of the art in the aggregated modeling approach has limitations, which under certain conditions can lead to an inaccurate assessment of system stability. It is worth noting that the said conditions are not hypothetical or rarely occurring edge cases but rather system conditions that could happen with low to medium probability, especially under high DER penetration scenarios of the future.
As such, attention has been growing towards a detailed modeling approach whereby individual D-system and DERs are represented, along with T-system, eliminating many of the challenges associated with the aggregated modeling approach. The detailed modeling approach represents the D-system in detail including high fidelity DER models, thus eliminating the need to accurately parameterize the simulation model for various operating conditions. Specifically, the detailed modeling approach may include a full spectrum of DER dynamics, including PLL, DC link dynamics, time dependency of protection components as well as logic for the operation of D-system components such as regulators, switched capacitors, etc. A comparison of the aggregated modeling and detailed modeling approaches is included in Table 1. The detailed modeling approach also brings in a number of practical development and usability challenges. This paper details the relevant information that helps address these issues from a practical standpoint.

B. EXISTING APPROACHES TO DETAILED T&D SYSTEM MODELING AND SIMULATION
The detailed modeling approach that models T-system and D-system in detail has received considerable interest over the last decade. Over the years researchers have presented different approaches towards a solution. The authors in [2] presented a Schur complement based approach and compared T&D results with the T-system only approach. A switching scheme that selectively switches those distribution systems that are affected by the disturbance on T-system, thereby reducing the computation required for detailed T&D system simulation is presented in [3].
The detailed modeling approach for T&D system simulations over the years has evolved into two schools of thought: • Combined T&D simulation approach where all the equations relating to both transmission and distribution system are modeled together within the same platform.
• T&D Co-simulation which takes established tools from each domain and interfaces them to work together.
Both approaches have their own merits and demerits. Without delving into which approach is better suited for T&D system simulation, the authors have attempted to list out some of the existing tools that are capable of detailed T&D modeling in Table 2. As can be seen from Table 2 both the combined and co-simulation approaches have been developed and used in recent years, which leads to the natural question, how do the results of T&D co-simulation compare with the combined simulation approach? The authors in [4] presented such a study for quasi-static time series (QSTS) analysis of T&D systems.
While the combined simulation approach provides the ease of using a single platform to develop the T&D simulation model, developing these components from the ground up requires an enormous development and validation effort. Thus, significant effort has been put into the development of the co-simulation approach which leverages the fact that individual T-system and D-system software and corresponding models have already established validity. It also provides ease to the transmission and distribution system operator in that they can use existing models in the simulation tools/software they are currently using without the need to remodel their system in a new software. The research question that needs to be answered to establish confidence in the T&D cosimulation approach is to answer the mathematical and software validity of interfacing the T-side and D-side. Interfacing of T&D co-simulation approaches can be broadly classified into loose and tight coupling schemes. Reference [5] presents a tight coupling scheme for T&D QSTS analysis. Reference [6] presents an analytical method for tight coupling. Reference [7] provides a detailed comparative study of T&D interfacing techniques. Approaches to T&D dynamic co-simulation presents additional challenges which have led to increased interest in this area. The authors in [8] researched the impact of time steps on convergence between T-system and D-system. Reference [3] presents a decomposition and localization approach for combined T-system and D-system analysis.
The goal of this paper, however, is to present the readers with the necessary information -from a practical power system T&D co-simulation perspective. Specifically to develop TDcoSim, an open source T&D steady-state and dynamic analysis tool. In addition, the authors wish to share the practical challenges faced during the development of TDcoSim and present its capabilities [9]. The rest of the paper is organized as follows: An overview of the T&D co-simulation is provided in Section II, Section III provides discussion about the various component modeling in T&D co-simulation, Section IV discusses how various simulators for different components are interfaced with one another, Section V discusses the overall co-simulation software development challenges and its associated modules. Conclusions are given in Section VI.

II. T&D CO-SIMULATION -A BRIEF OVERVIEW
There are two fundamental challenges with interfacing multiple simulators in a co-simulation framework: i) message passing, and ii) time synchronization. Irrespective of the methodology used, the individual atomic units of co-simulation are often run as separate processes. These processes need to communicate with each other in order to pass locally available information to other processes. In addition, these processes could be running within the same compute node or could be running on different compute nodes.
Passing information across the different atomic units of co-simulation is a necessary but not sufficient condition to ensure that the information is exchanged as needed. Time synchronization is the additional constraint that ensures the required information is available at the necessary time. Each atomic unit of co-simulation can run at its own internal time step dt i . Time synchronization between atomic units i and j occurs when, In this example, dt i > dt j and dt i is exactly equal to dt j after K time steps of the j th atomic unit. Indeed, this is a necessary condition for time synchronization to happen between two atomic units. If there are n atomic units in a co-simulation, the co-simulation engine/master/framework logic should be able to resolve the dependency graph for every time instance 't' of co-simulation.
As an example, consider a co-simulation involving three atomic unitsa, b and c, which has internal time step of 1, 4 and 10 seconds respectively. Let us assume that b and c requires input from a. At t = 0 all three units synchronize, then at t = 4 and t = 8, {a,b} synchronize. Units {a,c} synchronize at t = 10 while {a,b} continue to synchronize at t = 12 and t = 16. Both {a,b} and {a,c} synchronize at t = 20 and the process repeats itself until simulation end time is reached. The dependency graph in this example is a directed acyclic graph (DAG). Dependency graphs that are DAGs can be efficiently resolved. When there are cycles/loops in a dependency graph, the graph cannot be resolved without additional information.

A. CO-SIMULATION MATHEMATICAL CHALLENGES
Irrespective of the co-simulation methodology or interfacing technique, the nature of simulators and stiffness of the underlying equations make it such that only weak convergence guarantees can be given. The primary reason for this is that any information that is exchanged is truly communicated only after sub-problem convergence and not during VOLUME 10, 2022 sub-problem iteration. Let us take the case of two atomic units, where both are continuous-time simulators and are defined by differential-algebraic equations (DAEs). Each of these simulators uses an appropriate numerical integration scheme and time step dt and solves the resulting non-linear equations using an appropriate method such as Newton-Raphson. The information exchange across the atomic units occurs only after convergence at each atomic unit. This is in contrast to a combined solution/simulation approach where the entire set of equations are solved together and a boundary between the atomic units does not exist.
In the above example, the convergence of individual units is a necessary condition. However, it does not guarantee a global convergence, irrespective of whether a loosely or tightly coupled scheme is used. Essentially in a co-simulation even though each atomic units converge independently the interface may not converge. In the case of a tightly coupled scheme, when there is global convergence, the approach ensures a stable boundary condition. This implies that the co-simulation solution under these conditions guarantees that the solution will match with the solution obtained by combined simulation approach. So guarantees on solution accuracy can be given, when certain conditions are met and when a tightly coupled/iterative scheme is used. However, no strong guarantee of global convergence can be given. In other words, a lack of global convergence in a tightly-coupled scheme does not translate to a lack of convergence of the original problem when solved using combined solution approach. Nonetheless, empirical results have shown co-simulation approach in power system to be remarkably robust.

B. FRAMEWORK VERSUS TOOL
Co-simulation approaches, especially in power system, can be broadly categorized into two classes -frameworks and tools. For instance, HELICS [10] is a co-simulation framework while GridSpice [11] is a tool. A co-simulation framework allows any simulator to be integrated to the framework. Although simulator integration might be available, such integration is typically not part of the framework. Tools on the other hand are to be thought of as applications with interface to particular simulators. Hence, frameworks are extensible and tools are not, and tools have specific bindings that allow the user to simply configure and run co-simulation without creating a federate while such federates need to be created in a framework. TDcoSim is a tool rather than a framework.

C. LEARNING CURVE
One of the main challenges in software development is the ease of use i.e. the level of abstraction presented to the user through the software user interface should be such that the targeted user base will have a short learning curve. Yet, at the same time, the user interface should allow for sufficient customization so that the software is capable of being used for the majority of its intended use cases. This is especially challenging with co-simulation because of the multiple components involved and the exponential growth in simulation complexity. For instance, consider a 1000 bus transmission system where each load bus is replaced with one or more distribution feeders connected to the given substation. Depending on the size of the distribution feeder and the number of such feeders attached to a given substation, the total network size will increase by an order of magnitude two or three i.e. the 1000 bus T-only system will now be hundreds of thousands or even upwards of million transmission and distribution nodes.
The user interface should allow the user to easily set up, run and analyze such large systems -which is the fundamental challenge. Designing software for large-scale systems brings up a fundamental question -is the user going to use the software in a personal computer, cloud infrastructure, or in a high-performance computing (HPC) cluster. While a given software can be developed to work in multiple environments, it is still fraught with issues, simply because the developers do not control the target environment. As a result, performance cannot be optimized. In addition, the process of setup and running the software with the associated options, necessitates that the user has an adequate understanding of the underlying hardware and software setup -something that has a larger learning curve and is a deterrent to user adoption. Hence, TDcoSim was designed to operate on a personal computer, working equally well with the older python 2.7 version (32-bit) and the newer 3.7 version (32-bit and 64-bit) -the versions supported by PSSE. It has been tested on machines ranging from 4 GB RAM to 128 GB RAM, using Siemens PSSE versions 33 to 35. Furthermore, the user should not be required to know anything about the internal details of co-simulation. In the authors opinion, having such a requirement implies a higher learning curve which ultimately will result in lower adoption. In addition, if data is generated by multiple simulators where each simulator is producing data in its own format, then the data from different sources needs to be gathered and transformed into a new data structure that is easy to query and has high performance. It should be noted that if the network size increases drastically when going from T-only to T&D, so does the data size and complexity. Consequently, data management becomes a critical aspect. Hence, the guiding factors for a T&D co-simulation software are: • Lower learning curve; • Capability to scale to large T/D networks; • Support wide range of customization.

III. COMPONENT MODELING
In the combined simulation approach, we have one simulation instance solving the transmission system, distribution system, and DERs. In the co-simulation approach, the philosophy is that subsystems are simulated in dedicated simulators. Hence, we have two or more distinct simulation software instances that run concurrently with their own local simulation step size, and exchange information as necessary. The differences between combined simulation approach and co-simulation approach are illustrated in Fig. 1 where the gray boxes represent a unique computing process. Since each process can run in an unique core, the co-simulation approach is naturally amenable to parallelization across two or more cores. The specifics for each co-simulation vary. As an example, the different components of TDcoSim are illustrated in Fig. 2. The following subsections will further explain each component of TDcoSim.

A. TRANSMISSION SYSTEM SIMULATION (TSS) COMPONENT
The transmission system simulation (TSS) component used in TDcoSim is implemented using the power system industry's tried and tested software: Siemens PSSE [18]. Compared to the conventional transmission planning and interconnection study approach which utilizes the aggregated representation of loads and DERs in the distribution system at appropriate transmission-load nodes, the TDcoSim tool replaces the aggregated representation of distribution systems with actual distribution system models with loads and DERs VOLUME 10, 2022 connected at appropriate distribution system nodes and simulated using OpenDSS [19].

B. DISTRIBUTION SYSTEM SIMULATION (DSS) COMPONENT
The distribution system simulation (DSS) component used in TDcoSim is implemented using the open source distribution system modeling software: OpenDSS [19]. For each of the transmission buses where the loads are being replaced by the distribution system, the power output of a single distribution feeder is scaled up using a single scaling factor to match the rated load at the transmission bus and to simulate the effect of multiple feeders connected to a substation. The full library of traditional distribution assets (e.g. lines, transformers) within OpenDSS are accessible and used as-is. Further details on individual models used by OpenDSS can be found in [19].
Currently, the stable version of TDcoSim models each distribution feeder using a separate process due to non-reentrant shared objects associated with OpenDSS. There is however, an experimental version of TDcoSim that allows modeling multiple distribution feeders in a single process. This approach provides marginal improvement with regards to solution time, when the number of feeders within a process is chosen properly, as the number of context switches and communication overhead is reduced. The said approach provides major gains in terms of memory footprint as the libraries are loaded once in a single process for multiple feeders rather than once for each feeder across multiple processes.

C. DISTRIBUTED ENERGY RESOURCE (DER) SIMULATION INSTANCE
The capability to simulate individual DER instances within the distribution system model is one of the core capabilities of TDcoSim. Since grid-connected solar photovoltaic systems (henceforth referred to as PV-DER) account for majority of installed DERs, we chose to model them in TDcoSim. Additionally, only dynamic DER models are discussed here. Note that TDcoSim does have the flexibility to incorporate models of other types of inverter-based DERs like battery energy storage systems. A typical PV-DER consists of the photovoltaics module, a DC link capacitor, a power electronic inverter, and filters on AC side as shown in Fig. 3.
The circuit of an inverter branch consists of a series and parallel connected resistive, inductive, and capacitive elements. Due to the switching actions of the power electronic switches within the circuit, the inverter has non-linear dynamics. The dynamics of the primary and supervisory control system, smart inverter features, and protection logic during voltage anomalies should also be modeled.
The dynamics of the instantaneous value of the output current flowing through an inverter branch can be represented using the ordinary differential equation (ODE) (2), assuming an RL filter configuration.
where i u (t) is the inverter output current, and f u m (t) is the magnitude of modulation function at time 't'.
If the simulation explicitly models the instantaneous currents using (e.g. (2)), then it is referred to as the Electromagnetic Transient (EMT) model [20]. If instead we only model the phasor currents, then it is referred to as a rootmean-square (RMS) model [21]. Most simulation software uses either EMT or RMS models for modeling inverters. EMT models accurately represent both slow and fast moving dynamics while RMS models can only represent slow moving dynamics [22]. Due to the presence of high-frequency nonlinear dynamics, it is computationally expensive to simulate an EMT switching model. An EMT average value model, although comparatively an improvement still requires the use of integration time step in the order of tens to hundreds of microseconds. Since the goal of TDcoSim is to simulate thousands of DER instances that represent individual DERs, using EMT models is not a feasible option.
For simulating dynamic DER models, TDcoSim has two options: 1) A dynamic phasor DER model [23], [24]: available through the solar PV-DER simulation utility Python package. 2) A fast-DER model (RMS dynamic model): available within TDcoSim. For the studies that concern system operators and planners, the phasor based PV-DER model that captures slow moving dynamics is generally sufficient. Hence in TDcoSim, either a dynamic phasor model or RMS based fast-DER model is used to achieve a balanced trade-off between computational cost and added benefit to system stability analysis.

1) INVERTER DYNAMIC PHASOR MODELING
The rate of change of the phasor value of the branch current at a given time instant is given by (3) and is referred to as the dynamic phasor model. The advantage of a dynamic phasor model is that we can use a solution time step that is equal to one cycle of the inverter output waveform. For example, if we have a 60 Hz output waveform, we may be able to use a solution time step as large as 0.0166 s. The main advantages of using a dynamic phasor model are: 1) Ability to simulate a large number of DER model instances with less computational cost compared to an EMT model. 2) Simulate faster dynamics at or less than the waveform frequency (60 Hz).

3) Use a first principle model that is very similar in struc-
ture to the EMT model.
The following control components are modeled in the dynamic phasor model used in TDcoSim: 1) Primary controllers: Branch currents and PLL.
2) Secondary controllers: DC link voltage, active power, and reactive power. A detailed description of the DER dynamic phasor models can be found in [25]. The dynamic phasor models currently available in the solar PV-DER simulation tool supported by TDcoSim is shown in Table 3.
One of the observation that the authors made regarding the system of ODEs representing dynamic phasor models was that dynamic phasor models tend to be stiff in regions of discontinuities like faults and ramp-ups. It was observed that simulating more than 1000 DER instances would cause a drastic increase in solution time even when used with high-performance ODE solvers, which is why the authors invested effort in developing fast-DER models using first order dynamic equations.

2) FAST-DER USING FIRST ORDER DYNAMIC MODELS
The fast-DER model was developed to allow TDcoSim to simulate up to hundred thousand DER instances. The fast-DER model is a RMS dynamic model and is based on the widely used aggregated DER model, DER_A [26].
The fast-DER model is a non-stiff ODE system allowing them to be solved quickly by using explicit integration schemes like Forward Euler. Using the fast-DER model, we are able to obtain up to 10X speedup over the dynamic phasor DER model for co-simulations with more than 5000 individual DER instances.

3) SMART INVERTER FEATURES
The smart inverter features are common for both dynamic phasor DER and fast-DER models. They are implemented in order to closely mimic the specifications of interconnection standards like IEEE 1547-2018. Smart inverter features modeled in TDcoSim are: 1) Response during voltage anomaly: the DER model may either ride through, go into momentary cessation, or trip during a voltage anomaly on the low voltage or high voltage side. The exact behavior is determined by the magnitude and duration of the voltage anomaly. TDcoSim has the ability to provide custom ride through logic for DERs. 2) Voltage support during voltage anomaly: the DER model is capable of modulating the reactive power output, if setup to do so, during voltage anomalies to bring back the voltage to nominal levels.

IV. INTERFACE MODELING
Apart from the individual simulators themselves, the two most important components of a co-simulation program are 1) interface model (IM) and 2) orchestrator [27]. The IM specifies how data is extracted from each simulator and transforms data into a format that can be understood by the simulator paired to it. There needs to be an unique IM for each unique simulator pair. An example of interface model configurations corresponding to different T models and DER model pairs is illustrated in Table 4.
In the following subsection we will discuss the various aspects related to coupling between different simulators, available interface options and interface between TSS-DSS and DSS-DER as it relates to TDcoSim.

A. TIGHT COUPLING VS LOOSE COUPLING
Simulators need to pass information to each other at every synchronization point. If this information is passed exactly once during each synchronization point then the approach is called loose coupling. In the case of tight coupling, the information is passed one or more times until a convergence criteria is met. Hence, tight coupling ensures stable boundary conditions.
The trade-off between the two approaches is speed versus accuracy. Depending on the type of simulation, dynamics of interest and the time step used, choice between the two approaches can be made. In certain cases, it comes down to trial and error. TDcoSim uses tight coupling for T&D steady state co-simulation and loose coupling for T&D dynamic co-simulation.

B. INTERFACE OPTIONS BETWEEN SIMULATION INSTANCES
There are a number of options available to implement the interfaces between different simulation instances.
• Each TSS, DSS, and DER simulation instance has separate subprocesses: This option would require the most number of subprocesses as well as communication links between the TSS, DSS, and DER subprocesses.
• All DSS instances will be on one subprocess and all DER instances on another subprocess: This option requires the least number of subprocesses and two communication links. However, during each timestep, a large array has to be exchanged between the TSS and DSS and the DSS and DER subprocess.
• A hybrid approach where simulation instances logically close to each other are on the same subprocess. For example, each DSS instance and all DER instances associated with it will be on the same subprocess. The transmission network process takes the server role and spawns unique client processes for each distribution network.
To connect the TSS and DSS instances located on separate processes, TDcoSim uses TCP/IP based server-client model. Each process connects to the target engines to run the simulation (illustrated in Fig. 4). The approach in general scales well as the available CPU cores are well utilized. The overhead is in the form of context switches and communication.
It should be noted that the T-system simulation as well as each D-system simulation involves large number of variables. However, only the boundary variables need to be exchanged between the simulation instances as shown in Table 4.

D. DSS -DER INTERFACE
For the DSS-DER interface, TDcoSim uses the hybrid option described in section IV-B (illustrated in Fig. 5 where an unique subprocess is spawned for each distribution feeder referred to as feeder subprocesses. Within each feeder subprocess, we simulate a distribution feeder using OpenDSS as well as the DER models associated with that particular distribution feeder. Although the components are in the same process, the DSS and DER simulations use different numerical solvers to obtain a solution at each time step. The main advantage of this approach is the DSS and DER simulations have access to each other's variables since they run in the same memory space. Hence, the interface between them is simple and extremely fast. This is advantageous when there are hundreds of DER instances within the same feeder.
The data exchanged between DSS to DER models is shown in Table 4. Note that individual DER models don't have individual ODE solvers. Instead, the DER models are updated with the appropriate voltages from DSS nodes, combined into a single large system of ODEs, and then solved together.

V. CO-SIMULATION SOFTWARE DEVELOPMENT
This section describes the software development challenges for the co-simulation software. One of the main challenges is the computational power. The total number of buses to be simulated are n+m, where n is the number of buses in a T-system, and m is the number of buses in all of the D-systems. The total number of buses in D-systems increases as a function of number of feeders and number of buses in each feeder. Hence, increase in number of feeders increases m drastically. As a result the computational burden also increases drastically. Hence, high computational resources are essential to handling large network simulations that needs to be solved in reasonable time. The size of the system that could be simulated is memory-bound while the time it takes to solve is CPU-bound. TDcoSim provides the extensible capability with the layering of the components and network-based data exchanges. This is further discussed in section V-A.
The other major challenge revolves around the usability of the software. Multiple simulators are involved in a co-simulation, with each simulator having its own data structures, configurations and APIs. The challenge then is to provide an unified means to configure, run, collect the resulting data from each simulator and transform it into an unified schema that can then be queried. Ideally one wants to have a shorter learning curve i.e. an user who is adept at configuring PSSE and OpenDSS simulations should have little to no difficulty in setting up and running T&D co-simulation. To this end, TDcoSim leverages JSON files -a popular data format in the industry, to configure the co-simulation. A single configuration file allows the user to pass large number of native options to the underlying simulators. In addition, the same configuration file can be used to setup the co-simulation in an intuitive manner.
TDcoSim uses templating, and templates for different study types are generated by TDcoSim and the user only needs to update the value of some of the generated configurations. Thus, TDcoSim allows the user to setup and run the co-simulation without requiring the user to know about co-simulation or programming. The details of this approach are discussed in section V-B. TDcoSim is distributed as an open source Python [28] package. Hence, the user is able to install, modify and use it at no cost with the source code and documentation available in the GitHub repository. In addition, TDcoSim provides the dashboard [29] and data analysis modules to help users analyze the large amount of simulation data. Further details are provided in section V-C.
The final challenge is the continuous development of the software. The co-simulation landscape, in terms of available hardware, technology stack and interfacing are rapidly changing. In order for a co-simulation software to be relevant, improvements needs to be made over time. While good software practices make it relatively easier to integrate changes, the process requires time and expertise. This challenge, to an extent, can be addressed by open sourcing the software. Hence, TDcoSim is developed as an open source software.

A. SOFTWARE ARCHITECTURE
To achieve the flexibility and extensibility characteristics required for co-simulation, TDcoSim software architecture is designed with four layers as shown in Fig.6. Each layer has a set of modules to provide specific services. The user interface layer includes user interaction modules that provide command-line, visualization, and analysis services. The procedure layer includes the system interaction modules. Currently, TDcoSim has static and dynamic simulation procedures as defaults to organize the modules for the power flow and dynamic simulation respectively. The model layer includes the simulation model and interface modules that implement the API call and internal procedure to provide services as power system components. The third-party layer includes the connection modules for the external softwares. Currently, TDcoSim supports PSSE and OpenDSS.
To enable flexibility, the modules are connected by facade functions. Consequently, users can add and modify the new simulation procedures and models by inheriting the abstract module classes. With this approach, new simulation logic or models can be easily defined by reusing the existing user interface and third-party modules. Hence, when the new technology and modeling tools are available, TDcoSim can adapt by leveraging the reusable modules. For extensibility, TDcoSim defines data exchange protocols and synchronization procedures for network-based data communication between T&D system simulations. The approach can be extended to use multiple machines to simulate larger networks with the same simulation logic.

B. INPUTS AND OUTPUTS
The input and outputs for TDcoSim are designed to provide the user with a streamlined user experience. For the inputs, the T, D, DER, and co-simulation configuration parameters can be defined using intuitively named key-value pairs defined in JSON format. For T-system and D-system inputs, users can directly use the PSSE and OpenDSS input files. For the outputs, TDcoSim integrates generated data from TSS, DSS, and DERs into a single dataframe. Details on the types of the input and output files are listed in Table 5.

C. DATA ANALYTICS
TDcoSim produces time series data from each T-system, D-system, and DER node involved in the co-simulation. For a large T&D system, analyzing this data manually can be tedious. Often post-simulation data analysis requires the development of additional software/scripts on the part of the user. TDcoSim has an inbuilt analytics module. The user can use data analytics in TDcoSim either as a library or as a dashboard. The dashboard allows the user to run interactive queries in the browser. Some of the important features are:   • Data filtration: The user can filter out data at the scenario level, T bus level, and the property level. For example, bus voltage at T-bus 1 with voltage violations. In TDcoSim users can filter data based on multiple properties like node number, voltage, frequency, etc.
• Metrics: The following metrics can be extracted through the data analytics module, e.g. a) number of voltage violations, b) maximum and minimum value, c) postdisturbance recovery time, d) stability time, etc.
• Data visualization: This feature allows users to visualize the generated data using an interactive dashboard rendered in a web browser. The major modules include: (1) GIS module as shown in Fig.7; (2) Table module as shown in Fig.8; (3) Plots module as shown in Fig.9 and Fig.10; (4) Analytics module with pre-defined methods for extracting useful information from the co-simulation data. This enables the user to perform analysis without having to write separate scripts or load the data into another visualization software.

D. DATA VISUALIZATION
We illustrate TDcoSim's capability to plot variables of interest from multiple scenarios (where each scenario is a different co-simulation). To this end a simple system with one distribution feeder as detailed below is used. 1) T system: IEEE 118-bus test system. 2) D + DER: IEEE 123-node feeder.
3) DER penetration: 10 % of distribution system load. 4) Simulation configuration: A 1-second simulation during which a 3-phase fault is applied at bus 5 of the T-system 0.5 s into the simulation.  Here each scenario uses a different type of DER interconnection standard and hence they behave differently during the voltage anomaly. Fig.9 compares the impact on real power demand while Fig.10 compares the impact on the generator speed.

VI. CONCLUSION
The goal of this paper is to outline the necessary details required to build T&D co-simulation software -to enable better representation of complex interactions between transmission and distribution systems, particularly under high levels of DER penetration. To this end, the paper outlines the rationale, challenges, and implementation details of TDcoSim -an open source T&D+DER co-simulation software. The paper presents the modeling details, as well as computational and practical challenges. Specifically, the choice of interfacing techniques (loose vs tight coupling) and the trade-off between those choices with respect to convergence guarantees and computational effort. The paper also discusses an approach to maximize the number of feeders that could be modeled with the available RAM. The paper sheds light on DER models used in TDcoSim and their interfacing with distribution system. Details on synchronization of simulators are also presented. Finally, the solution implemented in TDcoSim to efficiently analyze large amount of co-simulation data through an interactive dashboard is also described. She is currently the Department Manager with the Department of Power and Energy Systems, Idaho National Laboratory (INL), Idaho Falls, ID, USA. Prior to joining INL, she was a Principal Engineer at the Argonne National Laboratory, where she focused on research in transmission and distribution systems co-simulation and planning and operations coordination, reliability impact of distributed energy resources on the bulk electric system, advanced distribution management system, and microgrid integration and impact analysis. She has had six years' of industry research and development experience, when she was a Senior Research and Development Engineer at ABB U.S. Corporate Research Center. Her current research interests include integrated energy systems, clean hydrogen production and grid integration, and energy systems decarbonization. He is currently an Energy Systems Scientist with the Energy Systems and Infrastructure Analysis Division, Center for Energy, Environmental, and Economic Systems Analysis, Argonne National Laboratory, Lemont, IL, USA. He has performed extensive research on distribution system modeling and analysis. His research interests include transactive energy systems, cyber-physical security, machine learning, statistical signal processing, game theory, stochastic modeling, optimization, and control theory. He is an Associate Editor of the IEEE TRANSACTIONS ON SMART GRID.
ROJAN BHATTARAI (Member, IEEE) received the B.E. degree in electrical engineering from Pulchowk Campus, Tribhuvan University, Nepal, in 2012, and the Ph.D. degree in electrical engineering from the University of North Carolina at Charlotte, Charlotte, NC, USA, in 2018.
He is currently working as a Grid Integration Program Lead and a Senior Application Engineer with the Chief Technology Office, GE Renewable Energy. Before joining GE, he was a Power Systems Research Engineer at the Idaho National Laboratory (INL) and before joining INL, he was a Postdoctoral Researcher at the Argonne National Laboratory (ANL). His research interests include renewable energy integration, transmission and distribution system modeling, and modeling and control of inverter-based renewable energy resources.
SHIJIA ZHAO (Member, IEEE) received the Ph.D. degree in electrical and computer engineering from Texas A&M University, in 2018.
He is currently an Energy Systems Scientist with the Energy Systems Division, Center for Energy, Environmental, and Economic Systems Analysis, Argonne National Laboratory, Lemont, IL, USA. His research interests include power system reliability and resilience, evaluation of power assets under extreme weather events, renewable integration, and related cost-benefit analysis.