A Versatile Software Defined Smart Grid Testbed: Artificial Intelligence Enhanced Real-Time Co-Evaluation of ICT Systems and Power Systems

In Smart Grid, the integration of Information and Communications Technology (ICT) systems and power systems has enabled real-time services and distributed controls, while the fusion of technologies necessitates a profound and versatile platform for the interdisciplinary research and evaluation. This article introduces a Software Deﬁned Smart Grid testbed architecture, which integrates real-world wireless communication systems and Artiﬁcial Intelligence algorithms to provide a re-conﬁgurable framework to meet different real-time Smart Grid testbed design requirements. Through prototyping and experiments, the architecture shows great potential in addressing co-evaluation challenges of ICT systems and power systems, as well as supporting real-time Smart Grid evaluations.


I. INTRODUCTION
The existing power grid is undergoing a fundamental transformation to the Smart Grid, which raises many challenges such as the penetration of renewable resources and the integration of smart devices.These changes also pose challenges on the Information and Communications Technology (ICT) systems, which are required to support the various Smart Grid services with different performance requirements [1].To provide a low-cost and flexible solution for the grid-wide information exchange, the industrial wireless communications technology is expected to play the key role in the emerging Smart Grid applications, especially for critical scenarios like Supervisory Control and Data Acquisition (SCADA) as well as geographically restricted areas [2].Moreover, wireless communication technologies in the fifth-generation (5G) The associate editor coordinating the review of this manuscript and approving it for publication was Yudong Zhang .systems have been expected to address many challenges in future Smart Grid, which include distributed voltage control, grid fault and outage management, precise load control regarding critical high-voltage direct current (HVDC) transmission faults, and the support for the automation protocols like IEC 61850 [3].Therefore it has attracted researchers from both the communication domain and power system domain to integrate wireless communication systems into the Smart Grid [4]- [7].However, unlike the wired communication based Smart Grid testbeds, most wireless Smart Grid testbeds are based on numerical results and simulations.There still lacks testbeds with real-world wireless communication systems as well as a generalized design method to integrate these wireless communication systems into the realtime evaluations [6].
As pointed out by the IEEE Task Force on Interfacing Techniques for Simulation Tools in [4], it is necessary to jointly consider ICT systems and power systems to design the entire Smart Grid system.Besides benefits such as flexibility and efficiency, ICT systems are also introducing new complexity and reliability challenges to the Smart Grid.This leads to a deeply coupled relationship between ICT systems and power systems, which necessitates a new versatile testbed to address both systems at the same time [5], [6].Unfortunately, existing testbeds commonly oversimplify ICT systems via rough estimations during the ICT-dependent control [4].New ICT technologies, such as the emerging 5G technologies, have made revolutionary changes to the wireless communication devices and algorithms [3].Many important changes are made in the Medium Access Control (MAC) and Physical Layer (PHY), which are the most oversimplified layers in the existing Smart Grid testbed design [4]- [7].To this end, a novel hybrid realtime Software Defined Smart Grid (SDSG) testbed architecture is introduced in this article.The motivation of this work is to propose a versatile Smart Grid testbed, which integrates real-world wireless communication systems and Artificial Intelligence (AI) algorithms to provide a re-configurable framework for different real-time Smart Grid testbed design requirements.Different from existing Smart Grid testbeds, the proposed SDSG testbed architecture integrates real-world wireless communication systems as part of ICT systems with full protocol stacks in MAC and PHY layers.The architecture can be reconfigured as different wireless communication networks and support advanced ICT technologies including AI algorithms.The architecture exploits three key design methods, including general-purpose hardware support, software defined modules design and modular design.These methods together provide new Smart Grid design solutions in addressing the cross-system interfacing and integration challenges identified in [4] and [5].
Recently, the advances of AI technologies have shown great potentials to address long-standing challenges in Smart Grid research, especially with the increasing system complexity, uncertainty and real-time big data [2].For example, both renewable energy resources and wireless communication channels are subject to fast-varying physical environments, which introduce uncertainty variations to the overall system.Meantime, the cooperation among the widely deployed smart meters and controllers not only complicates the grid control, but also puts pressure on the real-time communications with scarce spectrum.The AI algorithms, such as deep learning and reinforcement learning algorithms, are able to learn from history and adapt to future changes.However, the existing AI research in Smart Grid focused on theoretical studies, where there is still a lack of testbed support for real-time evaluations.Although [8] integrates ICT and smart grid, it ignores the potential of AI in the computing platform.In the proposed SDSG architecture design, efforts are made to provide support for the integration of AI algorithms from a framework level.
In the remainder of this article, we first briefly review the existing Smart Grid testbeds in Section II.In Section III, we introduce a novel SDSG testbed architecture with a hybrid of software and hardware components, which addresses the challenges of the real-time co-evaluation of both ICT systems and power systems.The architecture is then implemented into a lab-scale prototype for a specific real-time Smart Grid scenario in Section IV, where AI algorithms are integrated into the Smart Grid testbed with the MAC and PHY layer functions of the ICT systems.Real-time experiments are also presented, which demonstrate the proposed SDSG architecture's ability in the co-evaluations between ICT systems and power systems.Finally, the conclusions and further discussions are presented in Section V.

II. STATE-OF-THE-ART SMART GRID TESTBEDS
The concept of Smart Grid extends traditional power systems to a more versatile architecture, especially with the integration of renewable energies, advanced ICT technologies and smart devices.As these potential services require cooperation among different sub-systems, Smart Grid testbeds are expected to server as the intermediate method between theoretical studies and commercial deployments.The testbed's form varies with its consisting components, which can be generally categorized as hardware based, simulator based and hybrid testbeds.The state-of-the-art Smart Grid testbeds are briefly reviewed as follows.

A. HARDWARE BASED SMART GRID TESTBED
Hardware based Smart Grid testbeds provide an isolated research environment with necessary components of a real system.Its scale can range from a comprehensive laboratory to a large-scale power grid.Small-scale hardware based testbed forms a natural Micro Grid architecture.Typical examples are the Micro Grid testbed hosted at Zhejiang University [9], the testbed at Illinois Institute of Technology in Chicago [10] and the Smart Energy Integration Lab [11].Various studies can be performed using these testbeds, including fault controls and isolation mechanisms.
Full-scale hardware based testbed can support experiments in multiple research areas, such as the transmission domain and the distribution domain.A typical example is the Jeju Island in South Korea [12].The testbed covers the whole island and is capable to support five major research areas, including the smart power grid, smart renewable and smart electricity service studies [12].Multiple field demonstration sites are built in Ideal grid for all project (IDE4L) [13] across Europe, where the next generation active distribution networks are thoroughly studied from hierarchical control architecture, virtualization, aggregation and utilization of distributed energy resources aspects [14].There are also hardware based testbeds focusing on specific areas, for example cyber security [16], real-time power system control [17], Electric Vehicle control [18], distributed energy [19], [20] and the integration of power systems and communications systems [21].
Evaluations with these hardware based testbeds provide real performances and prototyping experiences to the researchers, which are necessary to verify the reliability of the services before commercial deployment in the real-world.
The practical benefit is at the cost of the physical infrastructure investment, where the scale and research problems are well defined during the design procedure.

B. SIMULATOR BASED SMART GRID TESTBED
By modeling the hardware counterparts at a higher level, the simulators can provide a simplified environment with more flexible architecture as compared to the hardware based testbeds.The simulators usually focus on specific areas of the Smart Grid.For example, as the Smart Grid integrates more smart devices, its cyber security has received much attention.To support such studies, various testbeds have been developed, such as the Virtual Control System Environment [22], Virtual Power System Testbed [23], intrusion and defence testbed [24] and Industrial Internet of Things testbed [25].In [26], a software defined utility architecture was proposed, where the software defined network/anything (SDN/SDx) method is applied to Smart Grid for the flexible network management and the cyber security enhancement.Different from [26] where the study is on the network layer and application layer in the ICT system, this work focuses on the technologies in MAC layer and PHY layer, as well as their integration with AI technologies and Smart Grid services.
As Smart Grid is an integration of different sub-systems, the co-simulation method has been proposed to study such coupled effects.The co-simulation method combines the power system simulator with the communication network simulator to approximate the real Smart Grid scenario.Examples are the Testbed for Analyzing the security of SCADA Control Systems (TASSCS) [27], SCADASim [28] and the Mosaik based testbeds [29].
The simulators exploit pre-determined models and finite parameter sets to characterize the complex real-world system, which is beneficial to functionality evaluations.Whereas the practicality of these testbeds highly depends on the feasibility of their models as well as their modeling methods, where the implied assumptions and simplifications can be challenging to justify in the real system.For example, most state-ofthe-art communication network simulators such as NS3 and OPNET are based on discrete-event driven model, which simulates the communication network as a sequence of events with no changes between consecutive events.But real systems are continuously changing with time, while synchronizing simulators from the communication system and the power system is a recognized challenge [4].Compared to cascading hardware devices, interfacing simulators is still not an easy task.This is due to the fact that these specialized simulators are not designed to support power system simulations, and vice versa.

C. HYBRID SMART GRID TESTBED
Taking the advantages from both hardware and simulator based testbeds, the hybrid Smart Grid testbed provides a trade-off option by implementing part of components in physical hardware devices, while others are simulated using simulators.The architecture of the hybrid Smart Grid testbed varies with its research focus, where typical examples are the ScorePlus testbed [30], the Network Intrusion Detection System [31] and GreEn-ER1 Industrial Control Systems Sandbox testbed [32].
In the Smart Grid, the power system forms the infrastructure, which also confines the research areas that a testbed can support.It would involve vast investment for the hardware based research in certain areas, such as transmission domains and renewable energies.Therefore real-time power system simulators, such as Opal-RT and RTDS, are favored by various hybrid Smart Grid testbeds.Examples are the cosimulator testbed [7], PowerCyber testbed [33] and the testbed developed by Texas A&M University [34].
The hybrid Smart Grid testbed combines the practicality of the hardware based testbed and the flexibility of the simulator based testbed, but it also inherits and even creates new challenges from both sides.For example, the Hardware-In-the-Loop (HIL) experiments have to control physical hardware devices via limited Application Programming Interfaces (APIs) from the simulators, while it also requires the accordance between the hardware and simulator components in the real-time.
The review of these state-of-the-art testbeds shows that most testbeds are project specified.The main reason is rooted in the purpose of Smart Grid testbeds, which is to serve as an experimental tool for the theoretical studies.The case-bycase design method has many recognized issues, such as the reuse of the developed resources for new projects and the comparison between different testbeds.In the remainder of this article, a novel and general framework for testbed designs is proposed to address these challenges.

III. NOVEL SOFTWARE DEFINED SMART GRID TESTBED ARCHITECTURE
Practical Smart Grid systems consist of both ICT systems and power systems, where the ICT system is a broad concept including both communications systems and signal processing systems.Most significant advances in 5G technology are in MAC and PHY layers, while 5G is also treating Smart Grid as its targeted application scenario [3].Currently there is still a lack of hardware level support for the MAC and PHY level integration study in Smart Grid, which is the main research gap addressed in this work.Due to the revolutionary development in 5G and beyond networks, the industrial wireless networks are competitive against their wired counterparts.Therefore the wireless deployment in Smart Grid has attracted stakeholder's engagement from academy, industry and standardization organizations, including the International Business Machines Corporation (IBM), CISCO, Institute of Electrical and Electronics Engineers (IEEE), National Institute of Standards and Technology (NIST) and International Electrotechnical Commission (IEC).However, existing hybrid Smart Grid testbeds focus more on the power system side, while the communication system is usually fulfilled by simulators such as Network Simulator(NS)-3 and Optimized Network Engineering Tools (OPNET).The performances of the Smart Grid services with real hardware based communication systems still lack supports from testbeds.To such ends, a novel SDSG testbed architecture is proposed with the hybrid of real communication systems and power system simulators, whose main contributions are summarized as follows: • Instead of dedicating to a specific project or research problem, the SDSG architecture focuses on a general framework to support different real-time coevaluation requirements of ICT systems and power systems.The combination of three key design methods, including general-purpose hardware support, modular designs and software defined modules, is a novel effort in addressing the interfacing challenges between different sub-systems, or even research domains.
• To the best of our knowledge, the proposed SDSG architecture is the first to integrate the real-time wireless communication systems into the hybrid Smart Grid testbed.More importantly, the architecture supports the implementation of different real wireless communication systems within the same testbed.This enables not only the selection, evaluation and comparison of different wireless communication systems for the same Smart Grid application, but also the integration of more advanced wireless communications technologies such as 5G and Cognitive Radio into the Smart Grid.
• This work is the first effort to support AI algorithms from the hybrid Smart Grid testbed design aspect.It provides novel methods to integrate AI algorithms with functions in ICT systems and power systems, especially from the aspect of real-time evaluation.This is an important complement to the current AI research in Smart Grid, where existing works are focusing on models, analyses and simulations while the real-world evaluation aspect is still under-addressed.

A. HYBRID OF ICT AND POWER SYSTEM COMPONENTS
The proposed SDSG architecture exploits the hybrid of software and hardware components from both ICT systems and power systems.Specifically, three platforms, including the Real-Time Digital Simulator (RTDS), Software Defined Radio (SDR) and Computing Platform, form the hardware basis of the architecture as shown in Fig. 1.Different from other hybrid Smart Grid testbed designs, the hardware devices are all general-purposed in their individual research domain, while the functions of each platform are defined by the implemented software and hardware modules.

1) REAL TIME DIGITAL SIMULATOR ENABLED POWER GRID SIMULATION
The SDSG architecture exploits RTDS as the power grid simulator.The RTDS is a general real-time power system simulation platform with a modular design, which enables the SDSG testbed with flexible interfaces and functions.For example, external devices can be connected via the lowvoltage and high-voltage interfaces of Gigabit-Transceiver (GT) Front Panel Interface (FPI) modules for HIL experiments [35], while GTNET module can support automation protocols such as IEC 61850.
On the RTDS, the power system can be defined by software with a wide range of choices, from the transient analysis of a grid component to the steady-state analysis of a largescale power grid.The most important feature of RTDS is its strict real-time performance, which simulates the power grid operation at a typical time step of 25-50µs.This feature is especially critical to the real-time co-evaluation between ICT systems and power systems, since the power system is required to interact with ICT systems in real-time within the SDSG architecture.

2) SOFTWARE DEFINED RADIO BASED WIRELESS COMMUNICATIONS SYSTEM
The wireless communications system is a general concept, where a wide range of choices are available for different Smart Grid application requirements, e.g., Zigbee for low power metering and cellular networks for wide area coverage in demand side management.The hardware implementation of these networks will benefit the testbed with practical evaluations, but it may also confine the testbed's research scope.This is because the specialized communication chips cannot be used for other communication networks, which results in the common challenge that Smart Grid testbeds are project specified and resources are hard to be re-used or extended for different projects.
To address this challenge, the SDR is exploited in the SDSG testbed, where the generalized Radio Frontend (RF) devices only convert baseband signals to broadcasting waveforms in the air.Instead of wiring chips, the specific wireless techniques are defined by software defined modules.Therefore with the same SDR RFs, the SDSG testbed can be reconfigured as different wireless communications systems, or even dynamically changes its configurations such as frequency bands, modulation methods and communications protocols.

3) ARTIFICIAL INTELLIGENCE ENHANCED COMPUTING PLATFORM
The computing platform is the core of the SDSG architecture, since signal processing is required for both ICT systems and power systems, e.g.transforming wireless waveforms back to messages and calculating control values based on grid measurements.To support software defined modules from both ICT systems and power systems, the general computing platforms are exploited, including desktops, laptops and micro-controllers.Similar to the RTDS and SDR, the general computing platforms provide computing resources and input/output interfaces, while their functions are software defined as required.
Specifically, the integration of AI algorithms in Smart Grid is supported by the general computing platform.For example, when dealing with uncertainties in Smart Grid, AI models can be trained with historical data and replace the empirical criteria for better decision makings [36].The AI algorithms such as reinforcement learning algorithms can dynamically evolve the models according to the real-time environment, which enables the Smart Grid services to adapt to different application scenarios.Therefore as a versatile testbed for the research of the emerging technologies, the computing platform supports the integration of AI algorithms via its essential open source and software defined design.

B. ADDRESSING THE CO-EVALUATION CHALLENGES
Although the three platforms are named by their fulfilled hardware devices, the enablers are the three key designing methods, including general-purpose hardware support, software defined modules and modular design.

1) GENERAL-PURPOSE HARDWARE SUPPORT
In the SDSG architecture, the hardware devices for both ICT systems and power systems, including RTDS, SDR and general-purpose computing platforms, are designed to use general-purpose hardware devices.The versatility of these general-purpose hardware devices enables a wide range of potential technologies in the testbed.Besides the feasibility to exploit existing works in each subsystem, this hybrid also enables new multi-system integration modules.In the testbed design, it is a common challenge to address the compatibility between simulators as well as devices.In such cases, part of the general-purpose hardware can be programmed as the interfacing bridge.The general-purpose hardware devices also fit well with the general tasks of the Smart Grid testbed.For research involving new technologies not supported by existing chips, tailoring the general-purpose devices could provide feasible solutions.Note that during new hardware developments, the implementation on generalpurpose devices is also a common procedure.For example, the communication networks based on the SDR are also real-world networks, whose software modules can be transformed to hardware-ready devices via Field Programmable Gate Array (FPGA).These features support not only the evaluation of cutting-edge technologies in the testbed, but also the further commercial developments based on the testbed.It worth noticing that the SDSG architecture has a wide range of off-the-shelf implementation options for specific co-evaluation requirements.For example, the SDRs can choose from high-end Universal Software Radio Peripheral (USRP) series for high-throughput low-latency services, or affordable HackRF series for smart metering via Zigbee.The computing platforms can be micro-controllers like MBED [47], System-on-a-Chip products like Raspber-ryPi or a laptop.

2) SOFTWARE DEFINED MODULES
By defining the functions using software modules with general-purpose hardware resources, the software defined modules can fulfill the same functions as their hardware counterparts.With the general-purpose hardware devices in the SDSG architecture, many components of the Smart Grid can be defined by software defined modules.This advantage can be used to change the interfacing requirements among devices to the cooperation between software defined modules, where the latter is much easier to cope with as compared to the physical device gaps in the former.The SDSG architecture further addresses this challenge by exploiting Python as the primary programming language and referring to open source developing methods and resources.Thanks to the open accessed feature in the open source, the software defined modules can be built upon existing works or projects.It also enables software defined modules in different subsystems to be programmed and executed in the same environment.Furthermore, this helps addressing the bottlenecks in traditional hybrid testbed design, where the interfacing among devices or simulators is subject to the availability of the pre-defined APIs.
The software defined method provides novel approaches for Smart Grid testbed design, especially for co-evaluation tasks where testbed components are from different subsystems.When interfacing different testbed components, it has great advantages to adapt software defined modules with open source support, as compared to cope with heterogeneous simulators and devices.For example, when studying power system operation under communication latency, the real-time Smart Grid operation requires the cooperation between all necessary functions, including grid monitoring and control.Although these are important supporting functions, wiring special meters and controllers into the testbed is a very complex task.Instead, the software defined modules can be used to aggregate the necessary metering and controller functions based on general-purpose hardware resources, which can be later substituted by their counterparts for extended studies.It should be noticed that software defined modules are essentially different from the pure software codes in the simulators.The software defined modules use software codes to control general-purpose hardware resources to fulfill the functions of their hardware counterparts.But the software defined modules are compatible as pure software codes.By using software defined modules as software codes only, the proposed SDSG architecture can be compromised as a pure simulator testbed and forms a co-simulation hybrid testbed.A typical example is the wireless communication part, where the physical RFs can be replaced with software models to transform the hardware-based networks into a communication network simulator.

3) MODULAR DESIGN
The SDSG architecture applies the modular design for its components in both hardware and software aspects.This enables more flexible co-evaluations of ICT systems and power systems.New features can be extended by integrating new modules or modifying existing modules.For example, the change of communication networks from Zigbee to WiFi can be fulfilled by implementing different protocol modules with the same SDR RFs.Moreover, the designs and implementations for communication nodes are reproducible and can be reused for larger scale communication network researches.Another advantage is that the modular design breaks the barriers between ICT and power system domains, which are transformed to the cooperation requirements between individual modules.Traditional hybrid testbed design method focuses on the interfacing between subsystems, whereas each subsystem is developed and maintained on its own.Due to the mutual dependency between software and hardware in the traditional testbed design approach, the modification of partial functions may affect the whole system, where reusing the developed resources could be very hard.However, the modular design in the SDSG architecture exploits the potential of full substitution between software modules and hardware counterparts.For example, it is infeasible to substitute the generator model with a real generator for HIL experiments in the offline based power simulators, but this can be achieved via high power interfacing modules on RTDS and external converter modules.More importantly, most modules on the RTDS, SDR and computing platform can be re-used.As the modules are self-contained with standard interfaces, the SDSG architecture facilitates the experiments with new models and their comparisons with existing models.
Especially, the AI algorithms are also treated as modules similar to the other software or hardware modules.Due to the modular design method, the real-time performance of the entire testbed is decoupled to the real-time requirement for every module.This, in turn, makes the input and output of each module operate in the real-time context, where the AI modules, as well as other modules can focus on their respective real-time performance.For the online-training and online-testing algorithms like the reinforcement learning algorithms, the real-time performance can be achieved by mature implementation methods like Finite State Machines and the use of high computing-efficient cores in C++.For AI algorithms such as deep learning algorithms that require long training time, the models can be trained offline and then implemented in real-time.The architecture can also support the offline training procedure.By replacing AI modules with the data collection modules, single real-time datum can be gathered into a large training data set for the offline training purpose.Moreover, as the modules are independent functional blocks, they can be designed and implemented by different sources.This is especially important to the integration of AI algorithms, since AI is a large research division and its commonly used modeling methods, as well as design tools, may not be compatible with the power system and ICT system research domains.With the modular design, the AI algorithms can be developed separately and integrated as a module.

IV. TESTBED PROTOTYPING AND REAL-TIME EXPERIMENTS
The proposed SDSG testbed architecture provides a general framework for the co-evaluations of ICT systems and power systems, which can be tailored for specific research problems.In this work, we focus on the MAC and PHY layer technologies in wireless communications, as well as their impacts on real-time Smart Grid application performance under wireless communication channel uncertainties.Specifically, the fusion of AI technologies into MAC and PHY layer algorithms are to be emphasized.To support such studies, a Micro Grid scenario penetrated with renewable energies is designed and implemented.An AI-enhanced real-time voltage stability enhancement application is studied, which demonstrates the real-time Smart Grid functions including real-time monitoring of grid conditions and command and regulation [15].A wind farm is integrated in the Micro Grid, which provides active powers to supply dynamic loads, as well as reactive powers to help maintain the grid voltage stability.The Data Acquisition and Actuation Module (DAA) module is used to simplify the functions of sensors and controllers, which periodically monitors the grid status via the RTDS interfaces, and generates a 10 bytes message containing grid status measurements every 50 ms.The messages are transmitted to the control centre through wireless communication networks, which then provides the reactive power control commands based on each measurement.The control command is then padded to 10 bytes and transmitted to the DAA modules for reactive power controls.Specifically, the AI algorithms are integrated to improve the wireless communications networks, while the impact of communication latency on the grid voltage stability performance is studied.The key parameters of the implemented prototype are summarized in Table 1.

A. HARDWARE IMPLEMENTATION
The devices used to prototype the SDSG architecture are illustrated in Fig. 2. On the RTDS side, the Micro Grid is simulated on two PB5 processor cards, while the grid status can be measured via the GTFPI module and the reactive power controls are accessed via the GTAI module.
Two laptops running Linux systems are used as the general computing platforms, which host the software modules from both ICT systems and power systems to emulate the Control Center and the Bus Node.The ARM MBED system is used as the distributed computing platform, which is software defined to a DAA module emulate sensors and controllers.
The RFs of the communication networks are fulfilled by USRP N210 supporting the frequency range of 1200 -6000 MHz.Two of them are used for communications between the Control Center and the Bus Node, while a third one is used to emulate an external user generating interferences.To synchronize between different devices and modules, the Global Positioning System Disciplined Oscillator (GPSDO) module Clock Distribution Accessory (CDA)-2990 is used, which provide universal and synchronized clock references with accuracy within 50ns.

B. PROTOTYPED SOFTWARE DEFINED MODULES
Following the SDSG testbed architecture design, the functions required for the co-evaluations are modularized as detailed in Fig. 1b.These modules are fulfilled by either or software modules.The frame structure in the prototyped testbed is shown in Fig. 3.In this section, some key software defined modules are detailed as follows.

1) SPECTRUM SHARING MODULE
Spectrum license fees could be a handicap for the application of industrial wireless in the Smart Grid, where a total of 190MHz spectrum resources could worth 1.3 billion pounds [37].This can be addressed by advanced wireless techniques such as spectrum sharing, which enables the Smart Grid devices to get access to the freely shared spectrum resources.In the testbed, the Cognitive Radio [38] is exploited as the spectrum sharing method to opportunistically access the unlicensed frequency bands for data transmissions.This module acts as a coordinator of the wireless communication systems as illustrated in Fig. 2b.Specifically, it will call the spectrum sensing module to listen to the surroundings, whose results are then fed to the channel selection modules for decisions.According to the selected channels, the spectrum sensing module will then apply a proper protocol from the protocol pool to format the baseband signals and configure the RFs for data transmissions.Upon the receipt of a message, the spectrum sharing module will call voltage stability enhancement module for control decisions or the Data Acquisition and Actuation module for grid control, depending on the messages and the node's task.The spectrum sharing module will then coordinate the protocol pool and RFs for replies.As these modules are all software defined modules, the coordination procedure can focus on interfacing modular functions without the need to distinguish between software and hardware components, or which subsystems they are belonging to.

2) PROTOCOL POOL
Modern communication systems are following layered design, where higher layers can work with different protocols describing the same lower layers.For example the wellknown WiFi, ZigBee and Bluetooth are protocols describing the same layers as the Ethernet protocols, which can be mutually substituted for different Smart Grid data transmission requirements.Contrary to the gateway design which aggregates parallel protocol specified RFs, the protocols are software defined to form a protocol pool on the testbed with the same SDR RF.Note that formatting the signals according to protocols is conducted in the general computing platforms via software modules, therefore the protocols can be changed during the operation on demand.
Different communications and signal processing technologies are integrated as sub-modules in the protocol pool, for example different modulation/demodulation, channel coding and channel estimation algorithms applicable to IEEE 802.11a/g/p protocols [39]are implemented in this testbed.Automation protocols like IEC 61850 are protocols for data link layer and above layers in the IEEE/OSI standard models, and they are compatible with lower layer protocols including the implemented IEEE 802.11a/g/p protocols, which defines the MAC and PHY layers.An example of mapping IEEE 802.11a/g/p protocols with IEC 61850 can be found in [40].With the support from the protocol pool, more choices are available to the AI modules to dynamically adapt the data transmission strategies with the instantaneous channel conditions.With SDR RFs, the minimal switching time overheads between protocols could be achieved on the level of 0.3 ms [46].

3) DATA ACQUISITION AND ACTUATION (DAA) MODULE
The research problem for the demonstration testbed is on the mutual impact between ICT systems and power systems.Grid monitoring and controller operations are essential to the whole experiment, but it is not the focus of this research.This is also a common issue when building a co-evaluation testbed, since the focused performances would require the full functions for evaluation and efforts are dispersed to the development of peripheral components.Taking advantage of the modular design in the SDSG architecture, the sensors and controllers are also treated as modules which are replaced by the aggregated DAA via MBED system.The grid monitor and controller models are implemented on the DAA module, which is also programmed to interface between the hardware modules on RTDS and voltage stability enhancement module on the computing platform.On request of the detailed operations of the sensors and controllers, part or whole DAA module can be substituted to its equivalent counterparts for more advanced studies.

4) VOLTAGE STABILITY ENHANCEMENT MODULE
During the grid operation, the voltages are also fluctuating with the change of system status, e.g. the connections of large loads or the reduction of wind power generations.Maintaining the voltage magnitude is one of the critical research topics in the power system stability studies, where out-ofrange values may damage the devices or even lead to systemlevel collapse.A voltage stability enhancement module is implemented on the testbed, which coordinates with the transmitter/receiver module to obtain the real-time grid status and command the reactive power controls.The module first runs the state estimation algorithm to obtain the whole grid status based on the received raw voltage and current measurements, and then optimizes the reactive power control via Newton Raphson method [41].These functions are built based on the open source solver PyPower using Python.Similar to other software defined modules, this power system module can directly call other modules on the testbed, e.g. the transmitter/receiver modules to read measurements or send control commands.
Algorithm 1 MAB Enhanced Channel Selection Algorithm 1: Initialize: Frame count f = 1, estimated mean reward vector r , where D is the discount factor indicating the importance of previous frames and R [t,f ] r is the instantaneous reward defined as the data rate at the t-th transmission round within f -th frame.

Adjustment Stage:
Calculate the adjusted mean reward vector r[f ] , as k is number of times the k-th channel has been chosen by now.

7:
Channel Selection Stage for Next Frame: a).Two channels associated with the highest r[f ] will be chosen as the selected primary channel and backup channel for next frame data transmission.b).The Control Centre and Bus Node exchange and update the channel selection information.

5) AI ENHANCED CHANNEL SELECTION MODULE
With the spectrum sensing module and the spectrum sharing module, the testbed is capable to dynamically access available channels for data transmission.However, channel selection for next transmission is still a complex and hard decision, since random selections [8] or energy detection [42] cannot address the challenges including the time-varying nature of the channels and the uncoordinated interference sources from other users.Therefore an AI algorithm, namely Multi-Armed Bandit (MAB) algorithms [43], [44], is implemented for the channel selection module.The MAB algorithm is fed with real-time spectrum sensing results, which are weighted against past observations.In this way, the dynamic features of the channel are learned, which is then used to make the empirical decisions on the next best available channels.The channel selection results in different performances experienced by the Smart Grid services, including the throughput, latency and the consequent voltage fluctuation.The real-time channel state information is used as the quantified parameter to reflect the channel selection result, which is fed back to the MAB algorithm to dynamically adjust its model for better Smart Grid communication performances as detailed in Algorithm 1.The real-time input and output of the AI modules are coordinated by the spectrum sensing module, while the algorithm in Algorithm 1 is implemented via high computing-efficient cores in C++ with a measured time performance about 10 −5 second.

C. REAL-TIME EXPERIMENTS AND EVALUATIONS
In this section, real-time experiments are carried out with the implemented hardware and software components.The aims of the experiments are manifold, including a) a systematical evaluation of the collaboration between the software and hardware components in the prototype, b) a study on AI performance gain with its integration into the Smart Grid systems, compared with traditional algorithms, and c) an evaluation of the impacts of real-time Smart Grid application due to real-world communication channel uncertainties.For this purpose, the key performance on AI algorithm, roundtrip communication latency and the overall voltage stability enhancement performances are presented as follows.

1) REINFORCEMENT LEARNING ENHANCED CHANNEL SELECTION
The MAB reinforcement learning module is exploited as the AI algorithm to enhance the channel selection performance during the transmission of measurements and control commands.This AI module is evaluated both in offline simulations and real-world experiments, whose results are presented in Fig. 4.Under dynamic channel conditions and unknown patterns of interference source, the integration of AI module shows advantages in selecting better channels for data transmission, as compared to the traditional energy detection based method [42] and random channel selection method [8].The experiment results agree with the general trend of the simulation results, which demonstrates that the testbed has fulfilled the expected AI integration tasks.Meantime, the comparison also shows the differences between realworld experiments and simulation expectations.One major reason revealed by the post-analysis is that the channel selection decisions are compromised by the lagged and imperfect channel status estimations in the real-world systems.Due to a VOLUME 8, 2020 lack of available models to characterize this close dependency between hardware and software components, simulations are following the common methods where the channel estimations are assumed to be perfect.It is a desirable feature for a testbed to support the modular-wise comparison between offline and real-time performances.The modules like AI algorithms can be developed and evaluated separately, or modified from open sources.The testbed can in turn benefit the researchers and developers in separate research domains, since they are facilitated to test and analyze their developed components in the joint ICT systems and power system scenarios.

2) ROUND-TRIP COMMUNICATIONS LATENCY
In a real-time smart grid application such as the studied voltage stability enhancement, the communication latency is a critical performance metric that could degrade the grid control performances, or even lead to a collapse.Every participating component will contribute latencies to the overall experienced delay, hence the round-trip communications latency is studied in the experiment as an end-to-end time performance metric.The round-trip communications latency is referred to as the latency between sending out the measurements and receiving the corresponding control command, whose histogram performance is presented in Fig. 5.For a hardware based Smart Grid system, the minimal latency is determined by the physical limit of all its hardware and software components.Knowing this extreme latency is critical, because any more stringent latency requirements will not be supported for real-world experiments, while more relaxed latency requirements (e.g., non-real time services) are potential to be supported via further development.Statistical analysis shows that the achieved average round-trip latency is 8.9ms, while the latency is ranging between 6.5ms and 11.8ms.This performance is very promising to support a wide range of Smart Grid application with the testbed, where a latency performance at the level of 10ms is expected [45].From a different aspect, the testbed can be also exploited as field measurement equipment for Smart Grid communication latency modeling purpose, where post-processing shows the round-trip latency is close to a normal distribution with a FIGURE 5.The round-trip time performance of the implemented prototype.

6.
Voltage magnitude performance at bus 2 of the micro grid.
standard variance of 0.0008.Unlike model-dependent simulators, this testbed can be also used to create new models based on the measurement and feedback to the simulationbased research.

3) REAL-TIME VOLTAGE STABILITY ENHANCEMENT
The voltage stability enhancement serves as a good realtime Smart Grid application example in the co-evaluations of ICT systems and power systems.To realize this Smart Grid application, it requires the cooperation of all modules listed in Fig. 1b, while the data flow between these modules is illustrated in Fig. 2b.The real-time experiment results in Fig. 6 indicate that the real-time voltage is maintained above the desired control threshold with the prototype, where the power system, the wireless communication system as well as the AI algorithms have cooperated to achieve this goal.This result also demonstrates that by following the SDSG architecture design, the prototype and its consisting modules are able to support real-time co-evaluations of ICT systems and power systems.Moreover, a comparative case study is also performed using the prototype, where the AI module is replaced with the traditional energy detection based module to fulfill the channel selection function.Due to the worse channel selection performance of the energy detection based module as analyzed previously in Section IV.C.(1), as well as its coupling relation with both power systems and signal processing systems, the overall voltage stability performance is degraded with two voltage violations.This study also shows that the testbed can be used as benchmark systems and support the comparison between different algorithms, which can be fulfilled by replacing corresponding modules.

V. CONCLUSIONS AND FUTURE WORKS
In this article, a SDSG testbed framework consisting of the RTDS, SDR and AI-enhanced computing platform is proposed.Specifically, its design methods in general-purpose hardware support, software defined modules and modular design show great potentials in supporting a wide range of real-time Smart Grid testbed designs.These design methods and the use of architecture components are illustrated via prototyping the architecture into a demonstration testbed.Realworld experiments on AI-enhanced voltage stability controls demonstrate that the proposed SDSG architecture is capable to integrate the components from both ICT systems and power systems to support real-time Smart Grid applications.
For the future work, the flexible SDSG testbed framework and the implemented prototype will be exploited to support further research topics in Smart Grid context, including the fully distributed schemes, diverse data generation sources and heterogeneous data transmission requirements.Regarding the RTDS, large scale power grid models will be considered, while the real power grid components such as energy storage, Photovoltaics (PV) and controllers will be connected to the testbed via RTDS interfaces.The existing RTDS models can be used to expand the study cases, such as Phasor Measurement Unit (PMU), SCADA, Battery Energy Storage System (BESS), thermal energy storage systems and distributed energy systems.Moreover, the protocol pool will be further expanded from both ICT systems and power systems where the internetwork layer, transport layer and application layer will be added into the protocol pool, which together with the already implemented MAC layer and PHY layer protocols to form the full TCP/IP protocol stack.Ethernet protocols, TCP protocols, IP protocols and UDP protocols will be added to the protocol pool to support the hybrid topology of wireless and wired communications.The automation protocols such as IEC 61850 will be included and implemented with the IEEE 802.11 protocol.More Smart Grid applications will be implemented on the power system side including both real-time and non-real time Smart Grid applications, such as the Demand Side Management (DSM) and metering.As for the Computing Platform, more AI algorithms such as deep learning algorithms will be integrated to address the uncertainties from both ICT systems and power systems, for instance, the renewable energy and load forecasting.

FIGURE 1 .
FIGURE 1.The proposed Software Defined Smart Grid Testbed architecture integrates components from both ICT systems and power systems.

FIGURE 2 .
FIGURE 2. The implemented prototype of the Defined Smart Grid Testbed architecture and the data flow among the testbed modules.

FIGURE 3 .
FIGURE 3. The frame structure in the prototyped testbed.

FIGURE 4 .
FIGURE 4. Simulation and experiment evaluations of the proposed AI enhanced algorithm against the random selection algorithm and the energy detection algorithm.

TABLE 1 .
Configurations for the prototyped testbed.
total number of transmissions T within a TX/RX slot, total channel number K .