An Open Source Tool to Compare Simulators on Large-Scale Cases — Application to Dynawo

Dynawo is an open source hybrid Modelica/C++ suite of simulation tools for power systems, geared towards the time-domain simulation of large transmission networks at different time scales, from the steady-state calculation to transient stability analysis. In this paper a set of open source tools is presented, designed for: (a) validation of Dynawo based on black-box testing against existing established simulators, or against previous validated versions of Dynawo; (b) exploration and analysis of the effects of modelling changes and model parameters on the network response, by means of A/B testing. In both cases the approach is based on the automatic generation of an extensive set of cases derived from a given base case, typically by means of N-1 contingencies. Most importantly from the point of view of power systems research, a concrete set of metrics is presented to deal with the practical problems of comparing different simulations of large, real-world systems. The tools, based on Python and Jupyter notebooks, are designed with flexibility in mind, in order to adapt the code to future testing scenarios and to extend the metrics used for comparing results.