On Modelling of Edge Datacentre Microgrid for Participation in Smart Energy Infrastructures

Datacentres are becoming a sizable part of the energy system and are one of the biggest consumers of the energy grid. The so-called “Green Datacentre” is capable of not only consuming but also producing power, thus becoming an important kind of prosumers in the electric grid. Green datacentres consist of a microgrid with a backup uninterrupted power supply and renewable generation, e.g., using photovoltaic panels. As such, datacentres could realistically be important participants in demand/response applications. However, this requires reconsidering their currently rigid control and automation systems and the use of simulation models for online estimation of the control actions impact. This paper presents such a microgrid simulation model modelled after a real edge datacentre. A case study consumption scenario is presented for the purpose of validating the developed microgrid model against data traces collected from the green edge datacentre. Both simulation and real-time validation tests are performed to validate the accuracy of the datacentre model. Then the model is connected to the automation environment to be used for the online impact estimation and virtual commissioning purposes.


I. INTRODUCTION
Datacentres are rapidly becoming a sizable part of the energy consumption of the society due to the increasing expansion of information and communication technologies. Over 3% (or approx. 370 TW) of today's generated electrical energy is consumed by datacentres [1]. To put this into perspective, the carbon footprint of datacentres is on par with that of the airline industry [2] and this rate of growth is only expected to increase. The so-called "Green Datacentre" [3] is a new type of datacentre that is being developed with the aim of building a sustainable future by utilizing renewable energy resources to reduce its carbon footprint.
There is growing interest in operating datacentres in Nordic countries due to their cooler temperatures, which can provide significant savings in the cooling cost and carbon footprint. According to NORUT [4], Nordic countries are surprisingly suitable for solar energy. However, due to the highly volatile and unpredictable local conditions, e.g., with frequent variations in cloudiness and the major differences between summer and winter, it requires other means of renewable resources to also play an active role in the production of electricity (i.e., hydro, wind). Sweden, for example, utilizes 54% of its total energy usage from renewable sources such as hydro and biomass [5].
Traditional datacentres operate as a consumer of the energy grid with backup energy sources such as diesel generators and Uninterruptable Power Supply (UPS). However, with the advent of the green datacentres and the integration of renewable energy resources (e.g., photovoltaics or PVs) in the datacentre microgrid, datacentres could become prosumers (i.e., an entity that is capable of producing and consuming power) and take a more active role in the energy community for the following reasons: Firstly, datacentres are often placed in the vicinity of residential areas, which can provide both opportunities and challenges in achieving higher energy efficiency of the society. The interplay between datacentres, residential and commercial energy users can be achieved by means of residual heat reuse for district heating, but also on shared use of renewable generation and energy reserve facilities. Implementation of such interplay requires developing technologies for tighter integration of datacentre control and automation with energy management systems of residential homes and industrial enterprises.
Secondly, renewable resources currently operate in a "fit and forget" approach, where the renewables are added to the current infrastructure rather than integrated. Typically, they do not play an active role in the energy distribution process: the existing centralized generation takes part in the controllability of the energy distribution, but the renewables do not. According to the "Smart Grid" vision, where the "prosumer" plays an active role in the electricity ecosystem, the renewables must be used more than now.
Lastly, datacentres are suitable for demand-response since they can be extremely flexible loads. For example, 5% load can be shed in 5 minutes and 10% can be shed in 15 minutes without changes to how the datacentre Information Technology (IT) workload (i.e., the consumption of the datacentre) is handled (i.e., via temperature adjustment and other building management approaches) [6]. In addition, datacentres are usually equipped with UPS systems, which typically have excess energy capacity and can be beneficial in the balancing of the electricity grid.
Therefore, datacentres equipped with renewable electricity production and energy storage can be an important active element in future energy systems. In most demand-response applications, it is assumed that the datacentre operators would adjust their usage, based on the grid operator's requests and driven by the corresponding incentives.
In the SONDER (Service Optimization of Novel Distributed Energy Regions) project [7], the goal is to investigate the viability of datacentres as an active participant (i.e., a prosumer) in the application of Demand-Response driven by energy market incentives. The first step in the project is to develop a complete datacentre model (in the context of the project, an edge datacentre) which incorporates PVs, battery storage, central grid, and the IT loads and is specifically adjusted to the simulation in the loop in connection with the real automation hardware and software, to be used for both virtual commissioning and online impact estimation purposes. Edge datacentre [8] is a small-scale datacentre situated close to the end-user that serves to reduce latency and improve customer experience. The edge datacentre that is modelled in the SONDER project has a small-scale microgrid that consists of a UPS capacity of 30 kWh, 10 kW of maximum power PV-cells and approximately 13.5 kWh total consumption from the IT-load and the cooling load. The simulation model is then validated against operational data generated by the real edge datacentre.
The paper is structured as follows. Section II surveys the related works. Section III presents an overview of the edge datacentre simulation model developed. Section IV presents the datacentre model in detail. Section V presents the case study scenario. Section VI briefly presents the control algorithm for the developed model. Section VII presents the software and hardware validation of the datacentre model in the loop with the automation platform. Conclusion and future works are presented in Section VIII.

II. RELATED WORK
There are existing research works, which aim to model different aspects of datacentres. For example, Tran et al. [9] have developed a simulation tool called Datacentre Workload Energy Simulation tool (DCWES) which estimates the power consumption of the IT and the cooling system. However, it is unclear how scalable the simulation model is. Besides, the DCWES does not consider the microgrid aspect of the datacentre.
Modelling of the datacentre's racks, pumps and chillers consumption provided by Mousavi et al. [10] and Berezovskaya et al. [11]. A datacentre toolbox presented in the latter article focuses on the modularity of the model, which can be used to construct a model of a datacentre of any size. The toolbox allows the creation of scalable server models that accounts for CPU consumption and server fan cooling. Similar functions but another approach for modelling of datacentre's cooling system is provided by Siltala et al. [12]. Here, the thermodynamic system was modelled in two different ways. The first is traditional (by using thermodynamics equations and laws). The second way is using the neural network capabilities for the calculation of the current results and for the prediction of the future values.
A fundamentally different approach to calculating the optimal amount of power flow between the main grid and green energy resources was proposed by R. Rahmani et al. [13]. In that work, the authors propose a new optimization model based on a radial motion optimization algorithm, in which the trade-off between greenhouse gas emissions and cost was a priority factor for research.
An approach for the energy management of a smart building based on SCADA systems was proposed by M. Kerami et al. [14]. In that work, the main target are the smart buildings with several types of load (critical, essential, and normal). The paper focuses on a control strategy and minimizing the power exchange with the main grid. The Modbus and Konnex serial protocol standards were used for communication between the control nodes. While some of these techniques could be used in the edge datacentre context, there are substantial differences between the two domains. From a modelling perspective, one can think of an edge datacentre as VOLUME 3, 2022 51 a large collection of small, identical hardware components, while the smart building architecture is more heterogeneous.
From an economic point of view, edge datacentres are bound by strict contractual obligations with their customers. For them, reliability, availability and low-cost energy sources are critical factors in their operation. Besides, the Matlab model mentioned in [14] is not publicly available.
Pelley et al. [15] expand the focus of modelling, which aims to model the power consumption of the whole datacentre. The presented model includes the interactions between the utility power, the cooling units, and the server racks. However, the model does not provide enough insights into the datacentre's operating states.
Zhabelova et al. [16] take this concept a step further by also incorporating the consumption down to the CPU level. In addition, Zhabelova's model allows for the simulation of the datacentre static operating states under various conditions.
A method to operate a datacentre with renewable energy that minimizes its dependence on grid power while minimizing capital cost was introduced by Arlitt et al. [17]. It was achieved by integrating datacentre demand with the availability of resource supplies during operation. One of the possible contributions here is in the possibility to use some newly designed methods to operate a datacentre with renewable energy and apply it to the virtual datacentre's model.
As seen from the observed related works, most datacentre models focus on the datacentre power consumption with models for estimating the IT consumption and the cooling system consumption. As far as the authors are aware, there's very little research work which models the interactions between the datacentre load and its microgrid entities. The intention of this research work is to investigate the potential of edge datacentres performing the role of prosumers in the regional energy market. Modelling tools used in the reviewed works lack extensibility and flexibility. The modelling framework proposed in this article aims to simulate the power profile behaviour of an edge datacentre. The simulation model is validated and adjusted with the data from a real edge datacentre. It models the power consumption of the IT-load, its interplay with the datacentre microgrid (i.e., UPS, PV, and the grid) and its ability to integrate with the real or virtualized automation infrastructure.
Therefore, the intended contributions of the paper are as follows: 1) A novel model which models the power dynamics of an edge datacentre microgrid consisting of the IT-load, renewable PVs, UPS battery and the grid connection. 2) A distributed microgrid controller which performs the role of the Energy Management System of the datacentre microgrid. 3) A hardware-in-the-loop (HiL) co-simulation platform for the validation of the datacentre model.

III. OVERVIEW OF THE EDGE DATACENTRE MICROGRID MODEL
The microgrid model is based on the edge datacentre that is operational at the RISE SICS ICE datacentres research centre  [18] (a partner in the SONDER project) in Luleå, Sweden and its structure is shown in Fig. 1.
The edge datacentre is structured in the following way: 1) The datacentre's servers are in the module Box. It consists of Central Processing Units (CPUs), organized in racks, with server fans performing the cooling of CPUs. 2) In terms of the power supply, the edge datacentre is powered by the central grid and the PV with a peak power of about 10 kW (poly-crystalline solar panels, 40 pcs., 250W@24V). The UPS (shown as a battery) is utilized as a backup power source (Lithium-ion batteries, 20 pcs., 12V@100Ah).
3) The cooling of the servers is achieved primarily by the Computer Room Air Handler (CRAH) units [19], with thermal storage as a backup cooling supply. 4) A centralized controller, which coordinates the operation of all these modular datacentre's components. The scope of the modelled edge datacentre is restricted to the UPS, the PV, the grid, and the IT-load consumption as shown in Fig. 2. Note that the total power consumption of the datacentre consists of both the IT consumption and the power consumption of the cooling system. In this paper, only the model of the IT-load is presented and the modelling of the power consumption for the cooling system will be part of future work.
The characteristics of the datacentre microgrid are as follows: 1) The microgrid operates based on the European standard of 50 Hz. 2) The maximum capacity of the PV_plant is 10 kW. 52 VOLUME 3, 2022 3) The maximum capacity of the Battery_plant is 30 kW. 4) The maximum power consumption of DC is approximately 13.5 kW. The model is developed in two steps. Firstly, the model was created using a collection of standard elements available in Matlab/Simulink. Then, the model was refined to track the real data.
The key parameters of the designed model are: 1) The amount of power consumption of the edge datacenter (proportional to the computing load), 2) The battery charge/discharge amount of power, 3) The amount of power, produced by PV, 4) The utility power production amount.

IV. MODELS WITHIN THE SCOPE OF THE EDGE DATACENTRE A. MICROGRID 1) PHOTOVOLTAIC PANELS
The solar panels used in the microgrid are equipped with a maximum power point tracker (MPPT) [20] to get the maximum electrical power from solar irradiation.
The solar panels of the edge datacentres have a maximum capacity of 10 kW and are equipped with boosting voltage block (which is boost the voltage level to 300V-400V). The total power produced by the solar panels can be calculated as [21] in (1): For this model, the most used solar panel type for industrial purposes with maximum output power equal 300 W, surface area -1.7 m 2 and μ ≈ 15% was taken as an example. The implementation of the "PV_plant" is shown in Fig. 3 and consists of the following: 1) An input signal "ctrl1" which controls the on and off state of the inverter. 2) An inverter that converts direct current to alternating current (DC/AC). 3) An active power measurement block that measures the active power, current, and voltage of the AC line.

2) BATTERY (UPS)
The UPS battery storage of the edge datacentre has a maximum capacity of 30 kWhs and consists of serial and parallel connection of Li-ion (Lithium Ion) battery. The power capacity can be calculated similarly to [22] in (2): The main purpose of the "Battery_Plant" block is to store excess energy. The charge and discharge cycle of the battery presented in [23] is described in (3).
The implementation of the "Battery_plant" is shown in Fig. 4 and it consists of the following: 1) An input signal "ctrl" controls the on and off state of the inverter. 2) An inverter (DC/AC).
3) An active power measurement block that measures the active power of the "ac_power", "ac_current" and the "ac_voltage" as output. 4) A "bat_mode" control signal which switches the charge and discharge state of the battery.

3) GRID
According to the Swedish Energy Agency [24], the top three sources of energy production are hydro, nuclear and wind. They all are synchronous generators with controllable amplitude, phase, and the frequency of AC current, which parameters can be described as in [25], shown in (4): During the work of the generator, Mn(t) should not be more than its maximum value as per (5):  If Mn would be greater (or even close to) Mmax, the generator will be out of sync and normal operation will be stopped.
The implementation of the "Grid" is shown in Fig. 5 and consists of the following: 1) "phase_A", "phase_B" and "phase_C" terminals, connected to the phase A, B and C of the main grid respectively). 2) Two controllable parameters for the synchronous generators are: a) "Pm" -the amount of mechanical energy that is applied to the synchronous generators b) "Vf" -exciting voltage level that is applied to the rotor coils To control the synchronous generators, the "generator control system" block was designed with the control algorithm shown in Fig. 6.

B. IT LOAD
The datacentre power consumption is mainly attributed to the two subsystems: IT and cooling. This work concentrates on modelling of IT power consumption. According to recent research, cooling systems consume around 40% of total energy consumption in modern datacentres [26] thus cooling power is approximately 60% of IT power. Therefore, the total power can be estimated to be IT power +60% of IT power.
Modelling the power consumption of the IT sub-system needs looking at the consumption at the server level. The main consumers here are CPUs and server fans. According to [27] the CPU power consumption can be estimated by a linear function of the CPU utilization (6). The server fan power consumption is modelled by a cubic function of the fan's rotation speed (7) based on the well-known fan affinity law.
As depicted in Fig. 1, the edge datacentre is presented by the module Box. The module is equipped with two racks each of which carries 38 Dell PowerEdge R430 servers. Each server is equipped with two Intel Xeon E5-2603 v4 processors.
The model of the Box module is constructed utilizing a modular Simulink toolbox, which is described in [11]. The toolbox contains the building blocks representing individual components of a typical datacentre such as CPUs and servers and enables the modelling of the datacentres of arbitrary configurations. Fig. 7 shows the model of the server which is utilized in the Box module. As this work concentrates on the datacentre power consumption, the server model provides the server power consumption as output. The server block calculates the power as the sum of CPUs power (6) and server fans power (7).
The input of the server block determines the utilization (or workload) of the CPUs on the server. As each server carries two CPUs, the Util input is a vector with two values. The 54 VOLUME 3, 2022 speed of server fans (RPM) is calculated by the controller, which is aimed at maintaining the predefined temperature of the CPUs. Fig. 8 demonstrates the model of one rack of the Box module. The model consists of 38 server blocks, each of which calculates the power consumption of the respective server. The output of the model is the total power consumption for the entire rack. The input to the entire rack model is the set of 38 vectors each of which is the input to the corresponding server block.
Finally, the Box model contains two racks thus the model calculates their aggregate power consumption and takes as input set of vectors with utilization for all servers in the racks. If all CPUs work on the same utilization, the set of vectors can be reduced to just one vector. Fig. 9 shows a comparison of the modelling results with real data obtained from the Box module in SICS ICE datacentre.
The modelled IT power is close to the real one. The gap between IT power and total power can be explained by the power of cooling units: the lowest values of the total power curve are obtained at minimum utilization of the cooling system, in this case, the gap is around 30%; the highest values of the curve are at the maximum cooling system utilization, in this case, the gap is around 70%. The modelling of the cooling system is the next stage in improving datacentre modelling. In this work, total power consumption was estimated at 1.6·IT_Power.

A. CASE STUDY SCENARIO
To validate the microgrid datacentre model, a case study was carried out in the edge datacentre showing the interplay between the datacentre load, PV, battery, and the central grid. The edge datacentre microgrid is powered by the following control strategy: 1) PV production has the highest priority to reduce the cost of electricity from the grid. 2) The second priority is the grid which will make up the difference in production from the PV.
3) The battery is used as a backup supply when the grid is disconnected. 4) The battery will charge from the grid only if there is a surplus from the PV. The data collection was carried out at the edge datacentre installation at RISE SICS ICE as shown in Fig. 1. The data collection was carried out under the following operating conditions: 1) Due to the extreme weather conditions in Northern Sweden (which affects the level of sunlight during the year).
To collect relevant PV data, the experiment was carried out during the Swedish spring at the end of May.
2) The length of the experiment was for 6 hours, and it was carried out from 10 am to 4 pm.
3) The initial setpoints of the edge datacentre are as follows: a) The thermal storage temperature of the cooler was set to 10 degrees b) The pump rate of the water cooler was set to 100% c) The fan speed of the CPUs was set to 100%, which means the CPUs were operating at 100% utilization d) The datacentre is connected to the national grid, the PV cells, and the UPS battery.  Fig. 10.
The case study scenario is as follows: 1) At T = 0hr, the datacentre load is powered by the PV and the central grid. 2) At T = 3.05hr, the grid is disconnected. The battery is in discharge mode and starts to supply the datacentre load along with the PV. 3) At T = 4.1hr, the grid is re-connected. The battery starts to recharge itself from the central grid. Based on the collected data, the following observations can be made: 1) On average, the overall power consumption of the datacentre load is at a constant rate of 13.5 kW since the utilization of the CPUs were set at 100% for the duration of the data collection process. Note that the oscillation that is produced in the P_Load is due to the powering on and off the compressor as it attempts to keep the temperature at the desired setpoint.  2) On average, 5 kW were produced by the PV It is worth noting that graphs, presented in Fig. 10, are based on the original data recorded in a real datacentre. The data will be taken as a reference point when the comparison with modeling results in further steps of this paper will be made.

VI. DISTRIBUTED AUTOMATION SYSTEM
In the field of substation automation systems, the most used hardware devices are Intelligent Electronic Devices (IED) and Programmable Logic Controllers (PLC) [28]. However, current IEDs and PLCs are designed based on a centralized control paradigm, which allows very little flexibility, scalability, and portability of the automation control software. These three features are essential as the energy grid becomes more distributed with prosumers and as such, the automation software must also be based on a distributed control paradigm.
IEC 61499 is the reference standard for the design of distributed automation software [29]. It has been successfully used for the implementation of distributed intelligent automation systems in the energy domain, e.g., [30]- [33]. IEC 61499 introduces object-oriented design artefacts called Function Blocks (FB). The internal logic of the FBs is implemented in the form of an Execution Control Chart (ECC), which dictates the execution of algorithms based on the incoming events, and the triggering of output events after the execution of the algorithms. The ECC and the GRID controller FB are presented later in Fig. 11.
For the purposes of model validation and initial prototyping, a simplified IEC 61499 decentralized automation application was developed, composed of five controller FBs: one for Energy Management Control and the other four for controlling the PV, UPS battery, Grid, and the datacentre. The MAIN_CONTROLLER FB implements the microgrid control of the Datacentre. The other FBs (BATTERY, GRID, PV, and LOAD) are providing services of collecting the power measurements and actuating their respective energy entities in the datacentre microgrid.
It is worth noting, that the distributed approach makes control software more transparent, self-sufficient, and flexible. This is achieved through object-based design, where software structure follows the structure of physical systems. 56 VOLUME 3, 2022   The controller of our microgrid model is implemented as a function block network, whose interface is shown in Fig. 12. 1 This function block network incorporates capabilities to control battery, PV, and grid-related devices at the same time, with component controllers of each device connected to the FB network in Fig. 3. Because it is a distributed control system, it is possible to physically deploy each control block to a separate device. The structure is flexible, so it can be 1 The corresponding artefact of IEC 61499 is called subapplication. adjusted for another configuration of a microgrid by adding or removing the corresponding control blocks.
In the overall application, the function block, implementing the control functionality, relates to FBs implementing a kind of driver for end devices as shown in Fig. 14. Their roles are as follows: r "MAIN_CONTROLLER". The common control procedures for the entire system are executed inside of this block (for example conditions for turning ON or OFF battery, PV or the Grid, fault state detection algorithm, etc.), r "BATTERY", "PV" and "GRID". An important characteristic for the battery, PV and the grid presented here (for example, for the battery it is capacity characteristics, amount of charge/discharge current, etc., for PV it is power profile, which depends on solar irradiation activity, for grid it is RMS voltage level, maximum power production level, etc.), The priority of power supply to the datacentre is managed as shown in Fig. 15. The EMS operates either in "GRID_CONNECTED" or "GRID_DISCONNECTED" mode. When operating in "GRID_CONNECTED" mode (triggered by the REQ event and grid-connected is false), it can operate in the following four states depending on the connection status of the "PV" and the "BATTERY": 1) When the PV and the battery are both connected, the PV is supplied, and the battery is in the charge mode until 80% capacity is reached. 2) When the PV is connected and the battery is disconnected, PV is supplied. 3) When the PV is disconnected and the battery is connected, the battery is in the charge mode until 80% capacity is reached. 4) When both the PV and the battery are disconnected, only the "GRID" is supplying. When the EMS system is operating in the "GRID_DISCONNECTED" mode (triggered by the REQ event and grid-connected is true). It can operate in the following four states: 1) When the PV and the battery are both connected, the PV is supplied, and the battery is in discharge mode. 2) When the PV is connected and the battery is disconnected, PV is supplied. 3) When the PV is disconnected and the battery is connected, the battery is in discharge mode. 4) When both the PV and the Battery are disconnected, no power is supplied to the datacentre. It is envisaged, that the decentralized component-based control application can be deployed to the distributed network of control devices.

VII. MODEL VALIDATION
A data-driven validation approach was used to validate the correctness of the simulation model by comparing the result of the simulation model with real data collected from the edge datacentre running the same case study scenario described in section V. The model was validated with both softwarein-the-loop (SiL) and hardware-in-the-loop (HiL). In the SiL validation, the edge simulation model described in Part IV were tested under simulation in Matlab Simulink and the IEC 61499 simulator NxtStudio on the PC. In the HiL validation, the same model was simulated in the real-time simulator OPAL-RT connected in a loop with a real PLC (running the control algorithm developed in Part VI). OPAL-RT and PLCs were connected physically with analogue I/Os (to transfer simulation data to the PLC) and digital I/ Os (to actuate the circuit breakers in the model). The following subsections A and B explain the experiment in more detail.

A. SOFTWARE VALIDATION
The first validation test was performed under simulation. The case study scenario in section V was simulated in MAT-LAB Simulink on PC and the simulation data were compared against the real data traces collected from the edge datacentre. Fig. 16 shows the behaviour of the battery where the blue line represents the real data traces while the red line represents the simulation data. This was expected as the simulation was carried out on a personal computer, which is not a real-time simulator. Figs. 17 and 18 shows the validation of the Grid and the PV data respectively and the simulation result closely correlates to their respective real data. The simulation results, presented in Figs. 16-18 clearly demonstrate the advantages of the developed model, but at the same highlight some limitations. For example, the small error gap between the estimated and real data (01-52%), along with fast feedback response of the designed model (<10 seconds for Figs. 18 and 19) proves the model's usability. But the relatively high noise (thick red areas) in Figs. 16 and 18 highlights deficiencies of the control algorithm, used to convert the correlate closely with the real data, although there is a ramp direct current generated by the battery and solar inverters to the alternating current: the control algorithm cannot identify the stable point for regulation of power making it "swing" near the optimal regulation point, alternately being in the mode of under-and over-regulation. "Grid" simulation result (Fig. 17) is clear from noises because it doesn't include inverter ("Grid" is already related to the alternating current type of source). The imperfection of the control algorithm for inverter applies limitations related to the dynamics of the power transfer between blocks in the model. The practical applicability range of the model is limited to the difference between minimum and maximum of produced and consumed power to not exceed 3-4 kW for 10-20 seconds interval.

B. HARDWARE-IN-THE-LOOP VALIDATION
The next step in the validation process was to validate the entire microgrid model in a HiL setup. The validation environment was built using real-time digital simulator OPAL-RT [34] connected in the loop with BECKHOFF CX-2030 programmable automation controller (PAC). 58 VOLUME 3, 2022 In this validation environment, the distributed microgrid control, developed according to the IEC 61499 standard, was deployed on the PAC while the datacentre microgrid model was running on the OPAL-RT real-time simulator. The interfacing between the OPAL-RT real-time simulator and the PACs is implemented via physical (digital and analogue) I/Os as shown in Fig. 19. The HiL validation environment is shown in Fig. 20, which consist of the PAC, OPAL-RT simulator, and user PC.
The developed validation environment allows for developing and testing the flexible and decentralized automation logic using the IEC 61499 standard, which can later be deployed to the distributed network of heterogeneous automation devices, which is seen as the ultimate goal of this research and development, although detailed discussion of the automation logic is outside of this paper's scope.
To match the sent and received analogue signals between the PAC and OPAL-RT, the gain blocks were used in the OPAL-RT project to make signals suitable for emitting, transferring, and reading by PAC. A gain block was introduced in the datacentre model and parametrized according to (8) Here the A max_signal variable equals the maximum level of the simulated signal, calculated in HiL (red lines). For Fig. 21, for example, this value would be about 16000, for Fig. 23 it would be about 9000 and so on. A max_PLC would always be equal to maximum voltage resolution for the PLC's analogue inputs. In the case of the current experiment, the authors took extinction blocks for the PLC with the maximum signal level   equal to 10 volts. It means that in the case of the current paper, the A max_PLC variable is equal to 10. Fig. 21-23 show the real-time validation result of the battery, the grid, and the PV models respectively. To calculate the errors between the real-time simulated data and the real data, an average error is calculated according to (9): Here A real and A calc it is an amplitude of different lines (in Fig. 21-23 blue and red lines respectively) taken at some moments of time between 0 and 6 hours. If N steps will be close to infinity, then more correct will be the average error value. In this case, the simulation step was 20 and the comparison shows, that the resulting average error of calculation is between 9 to 15%. In comparison with simulation results achieved in the previous step (Part A. Software validation), the difference between the accuracy of calculated results is up to 10% worse than simulation results, obtained from OPAL-RT. These results were achieved partly because of the simulated model complexity and because the speed of simulation has been increased more than 1000 times to keep the real-time simulation characteristics. ARTEMIS capabilities, provided by OPAL-RT, are used here as an instrument for the acceleration of real-time simulation. This acceleration is achieved by an enhanced engine driver which estimates the calculation load and uses an optimal calculation method for different moments of time. It means that during the time, when calculation load will be bigger than some limit (depends on hardware and software versions of OPAL-RT and ARTEMIS) the ARTEMIS engine activates the approximation algorithms to keep the calculations' speed on the same level. Because of that, the big error between the calculated results and real data can be detected (especially in Fig. 23). While the error reduction is planned to be addressed in future work, the main benefit of the proposed two-step approach: 1) software validation and 2) HiL validation is the dramatic reduction of the system development time. As was demonstrated in sections VI and VII real control system design can be implemented along with the design of validation model with acceptable accuracy of the results.

VIII. CONCLUSION AND FUTURE WORK
This paper presents a novel datacentre microgrid model developed in MATLAB Simulink for the purpose of evaluating and validating distributed intelligent control of edge datacentres. The model is specifically aiming at the emerging type of edge datacentres. The edge datacentre model was tested and validated against real data collected from an experimental edge datacentre from the RISE SICS ICE datacentre research centre.
The microgrid model structure is also based on the real edge datacentre which consists of a datacentre load connected to the central grid, UPS battery and PVs in a microgrid setting. A case study consumption scenario was developed to generate data traces from the real edge datacentre to validate the simulation model.
Two types of validation tests were carried out. The first was a software validation test where the resulting simulation data was compared against the real data. The simulation results closely correspond to the real data, albeit with simulation lag during switching. The second was a HiL validation test between the real-time simulator OPAL-RT (running the datacentre simulation model) and a PLC (running the distributed microgrid control). The HiL validation improves on the results of the simulation test with an average error of calculation between 9 to 15%.
Moreover, validation in the loop environment was created based on the IEC 61499 architecture, as the platform for implementation and validation of intelligent smart energy asset functionalities.
For future work, the first step is to integrate the cooling model into the datacentre microgrid, to have a comprehensive simulation model of the entire edge datacentre. The second step is to simulate the datacentre microgrid in connection with the regional or even national grid model, with the integration of various renewable farms, for the purpose of co-simulating demand/response applications. These steps will be accompanied by the development and validation of intelligent energy management strategies to turn the edge datacentre into a smart energy asset of the societal energy supply infrastructure.