Control-Centric Living Laboratory for Management of Distributed Energy Resources

Variability and uncertainty of renewable distributed generation increase power grid complexity, necessitating the development of advanced control strategies. demonstrates a real-world testbed and the implementation of control strategies on it to mitigate the challenges associated with variability and uncertainty of renewable distributed generation. This control-centric testbed includes 4.6 MW of controllable building loads, a 1 MW solar array, and a 125 kW / 130 kWh battery energy storage system (BESS). The capabilities of the testbed are illustrated by highlighting the implementation of three specific scenarios relevant to future smart grid infrastructures. In the first scenario, photovoltaic output variability is mitigated with the BESS using adaptive moving average and adaptive state of charge control methods. The second and third scenarios demonstrate peak load management and load following control to manage uncertainty using the Intelligent Load Control (ILC) algorithm. The ILC modifies controllable loads using a prioritization matrix and an analytical hierarchy process. The three scenarios all operate at a different time-constant, and are each effectively addressed, demonstrating the versatility and flexibility of the presented testbed. This demonstrated ability to rapidly test the efficacy of alternate control algorithms on a real system is crucial to the maturation of future smart-grid.


I. INTRODUCTION
T HE electric grid has served society well for the last century, but the challenge of addressing climate change requires modernization of the grid to accommodate green energy sources and reduce air pollution emitted by traditional fossil fuel-based generation. This can be achieved by increasing the use of distributed energy resources (DERs) such as flexible loads, storage, and renewable generation connected to the grid at the distribution level. In addition to lowering costs for electricity consumers, DERs can provide services to the grid by providing distribution-level flexibility. Most importantly, it benefits society by reducing the CO2 emission impact of the power system. DERs, however, present numerous control challenges, requiring sophisticated treatment with dynamic control strategies. Renewable DERs, in particular, can severely impact the stability of the grid [1], due to their variable and uncertain natures. Their management is thus essential to overall system reliability.
Variability of renewable generation means continuous and rapid fluctuation of power generation due to cloud cover, or available wind [2]. Uncertainty means unpredictability of magnitude and timing of the power generation due to unforeseeable weather conditions [2]. As the impact of variability and uncertainty differs depending on the timescale, the time constant of the control strategies for mitigating those impacts differ [3]. For example, the time constant associated with variability in PV power is on the order of a few seconds; this can lead to voltage fluctuations that can significantly affect the stability of the electric grid if not compensated for by other resources. One strategy to minimize the variability is to use an energy storage system [4]. Uncertainty, meanwhile, due to the dependence of renewable generation on meteorological conditions, can lead to hourly or daily supply-demand imbalances on a much larger timescale than the variability, resulting in frequency deviation [5], [6] and voltage instability [7], [8] in the grid. These effects can be mitigated by demand response [9], [10], peak load management [11], or load following control [12].
Overall, these challenges can be reduced by scheduling and controlling the supply and demand of the power system in multi-time intervals [13]. Flexibility in power systems is defined as the system devices' ability to adjust their power consumption to accommodate expected and unexpected imbalances in supply and demand [14]. To study flexibility in the system and build planning models by analyzing their effect, a 213-bus test system is developed in [15]. In addition to planning, it is also necessary to actuate flexible loads and storage located on the distribution side of power system using advanced real time control methods. Researchers are developing various novel control methods for managing such devices. In [1], an optimal voltage strategy is developed through residential demand response to meet voltage control requirements of the grid in multiple timescales. Additionally, a novel reactive power control for a wind-diesel hybrid system is presented in [16], and mitigation of PV power fluctuation using moving average and state of charge control is presented in [17]. However, developing and validating practical test cases to evaluate feasibility with real system devices is a significant barrier to adopting such control strategies in existing power systems.
Validation approaches can be categorized as simulationbased tests, lab-based tests, living lab/real-world/field tests, or some combination of these approaches. Validation efforts frequently use software-based simulations for simplicity and flexibility. For example, a large scale testbed as a virtual power grid has been developed for validating closed-loop control technologies [18]. However, a pure simulation-based approach can lack the fidelity of testing on actual equipment. Design and implementation of a laboratory-scale, smart-grid test-bed is presented in [19], which is intended to foster the validation, using real hardware, of developed technologies.
Many lab-based testbeds have been developed for managing the variability and uncertainty of renewables. A labbased setup designed for testing smoothing strategies for solar power fluctuation has been demonstrated, using an energy buffer unit in [20], and using building thermal loads in [21]. A lab-level, microgrid test bed has been implemented to study real-time control for demand management and for energy price optimization [22]. These lab-based testbeds provide more compelling results than mere simulation. Other labbased testbeds combine hardware systems and software models with hardware-in-the-loop (HIL) simulation. For example, the authors of [23] built a HIL simulation testbed which develops measurements, control, modeling and actuation techniques for a national grid with a high DER penetration. A hardware-in-loop setup is employed for testing various control and optimization methods [24]. The development of a multi-site evaluation platform with advanced distributed management systems is presented in [25]. This multi-site platform can evaluate distributed control algorithms before field deployment for a smooth transition from a centralized to a distributed system.
Design of future smart grid operating capabilities will further benefit from the use of real-world scenarios as a means of validation. Advantages and disadvantages of a futuristic, real-world testbed on a university campus are detailed in [26]. Though real-world testbeds are expensive and difficult to implement, their results offer a higher degree of confidence relative to laboratory experiments or simulations. One such example is the International Hellenic University Testbed [27] which contains smart grid elements such as energy monitoring and management of smart buildings. The team has used the testbed to reduce the total daily consumption of a typical university office by approximately 16% [27]. The system does not, however, use real-time control for its evaluation capabilities.
Control strategies are a key discipline within the smart grid and microgrid paradigms [28]. Due to the dynamic nature of DERs, it is necessary to develop a testbed that can perform and validate novel control algorithms on real-world equipment. Despite the need for control-centric testbeds, few exist, and these differ in strengths and intended applications. The Smart Grid Testbed [29], developed at the Singapore University of Technology and Design, evaluates smart grid network design and control mechanisms. It has been used to monitor and record end-user energy consumption in real-time to design incentives and control algorithms for residential and commercial buildings with a focus on demand limiting strategy with daily constant target. A real-world building energy management system (BEMS) with a 5.76 kW/11.52 kWh battery for peak-load shaving applications is presented in [30]. The battery is used for demonstrating optimized, controlbased peak demand reduction with the BEMS. Canadian Renewable Energy Laboratory (CANREL), presented in [31], is a physical simulation tool for the design, development, and performance testing of islanded and grid-connected microgrid projects. A stochastic optimization-based algorithm was developed and implemented for peak load reduction using a commercial building test bed with a 1 MWh / 200 kW battery in [32]. VOLUME 10, 2023 The LAMBDA Lab MG testbed [33], located at the Sapienza University, is comprised of DERs for performing demand response strategies for mitigating uncertainties associated with renewable generation units (PV systems) and electricity loads. Two types of controls were developed in this lab: one for energy optimization of the local sources and the second for operating in the island mode during emergency conditions. Another control-centric testbed is the Jeju Testbed. This large-scale deployment is set up for both residential and commercial spaces at the Jeju Island in South Korea [34]. The Jeju Testbed leverages real-time feedback and renewable DERs to optimize energy usage. Its main focus is to test various market-based strategies for managing power consumption and advanced technologies to support new power grid elements, such as renewable generation, electric vehicle charging framework, and automation transmission and distribution networks. Furthermore, in Singapore, microgrid test beds are being implemented in the Eco Campus and Semakau Island microgrids with a focus on enhancing economic operation and maximizing the reliability of the power supply [35] The Clean Energy and Transactive Campus (CETC) project at Pacific Northwest National Laboratory (PNNL) has created a multi-campus testbed formed by connecting PNNL, the University of Washington, and Washington State University. It is used to develop transaction-based energy management solutions [36] and provides a platform for research and development on building-to-grid integration and information-based energy efficiency. These market-based or transactive-control-based power management and demand response techniques will help to mitigate uncertainty of renewable energy. These real-world testbeds have demonstrated hourly to daily market-based, demand management strategies using DERs. However these testbeds have, thus far, lacked the capability to perform sub-second timescale control.
The main objective of the paper is to present a controlcentric framework and its development in a testbed at the Scott Park campus of the University of Toledo. In order to demonstrate the testbed's capability, the paper presents an implementation of control strategies for managing the effects of variability using a battery and the effects of uncertainty by use of controllable building loads. These are sufficiently different algorithms to highlight the diverse capabilities of the system but are not intended to be an exhaustive list of the algorithms of which the testbed is capable. Relative to previous work, the Scott Park Testbed incorporates various loads and DERs, which not only enable researchers to observe real-life cause and effect, but also provide a unique degree of flexibility to test control algorithms for managing DERs. With rapid prototyping and analysis of novel control strategies, the Scott Park Testbed provides the ability to mature the field by determining best practices for management of future smart grids. The contributions of this paper are summarized as follows: 1) Discussion of an interoperable, scalable, distributed, and control-centric framework and of infrastructure development using the framework with the VOLTTRON platform [37]. The testbed is capable of full-scale, realworld control experiments that have real interaction with occupied buildings and provide real-time data monitoring and storage. 2) Description of the implementation and validation of a novel PV power fluctuation mitigation using adaptive moving average and adaptive state of charge control methods for sub-second level control. 3) Illustration of the implementation of peak load reduction with constant target and load shaping in compliance with variable 10-minute to hourly schedules using intelligent load control (ILC). The remainder of this paper is structured as follows: Section II describes the characteristics of and rationale for a control-centric framework. Section III shows the implementation of the control-centric testbed for DERs by using VOLTTRON. Section IV illustrates the capabilities of the testbed through a discussion of three control experiments conducted on the Scott Park Testbed: PV variability mitigation, peak load management, and load shaping. Section V discusses future control experiments with the testbed. Conclusions is provided in Section VI.

II. CONTROL-CENTRIC FRAMEWORK
A control-centric framework is developed here to prototype novel control strategies for the management of DERs. This framework consists of four layers: the device layer, communication layer, database management and monitoring layer, and control layer, as illustrated in Fig. 1. Each layer performs distinct tasks and operates on its own time constant in the process of performing the desired control operation: Device Layer (Smart Grid Objects): The device layer is the foundational layer and comprises the deployment's equipment; e.g. building automation systems, generation, and storage. This layer also contains microcontrollers, processors, communication modules, and local memory storage which connect the devices to the communication layer.

Communication Layer:
The communication layer connects the components in each layer. It must communicate high volumes of data efficiently between distributed components and must deliver commands reliably, as these qualities are crucial to the scalability of the deployment and flexibility of the control schemes. Cybersecurity measures can be implemented and enforced at the communication layer. Data Management and Monitoring Layer: The data management and monitoring layer provides data storage, real-time monitoring, and information retrieval. Additionally, user management and user authentication are accomplished at this layer. Remote services can be polled for data to be used by other components of the system. Control Layer: The control layer leverages the abstraction of devices, data, and information services provided by lower layers of the framework in an application programming interface which allows rapid and straightforward implementation of control algorithms. This separation of concerns allows developers to focus solely on the specifics of the control methods being tested.

III. IMPLEMENTATION OF THE CONTROL-CENTRIC FRAMEWORK AT THE SCOTT PARK CAMPUS
This section explains how the control-centric framework has been implemented at the Scott Park Testbed using VOLTTRON.

A. DEVICE LAYER
The device layer consists of eight campus buildings (4.6 MW), two solar fields (1 MW) and a 125 kW/ 130 kWhBESS which are connected to the local distribution network as shown in Fig. 2. This section details the infrastructure and modifications in the devices for integrating them with communication and control layers.

1) BUILDINGS
The Scott Park campus is controlled by a Siemens APOGEE Building Automation System (BAS). APOGEE is a proprietary system that comprises three levels of devices. At the lowest level are a set of serial networks of up to 32 zone controllers referred to as a Field Level Network (FLN). Each zone controller runs one of several applications providing autonomous PID control for a specific class of equipment, e.g. a variable air volume (VAV) box, unit ventilator, or fan coil. The devices on each FLN are coordinated by a field panel using the proprietary P1 serial protocol. The eight field panels at Scott Park each manage one building and may have up to three FLN buses. Using Siemens parlance, these panels constitute an Automation Level Network (ALN), directly managing equipment affecting multiple zones, including air handling units (AHU) and chillers. At the highest level, the Management Level Network (MLN), a software system called Insight coordinates the field panels over the Ethernetbased P2 protocol and provides facilities staff with a centralized control point for all the building HVAC systems. The load of the buildings and campus is monitored by seven Square-D CM-2000 meters. Four of these manage one building each. Two meters are used to monitor two buildings each. The last meter tracks power and energy use for the whole campus at the utility interconnect. As the testbed is used to evaluate and deploy algorithms related to building power, six VOLTTRON nodes, each corresponding to one building meter, are installed on Raspberry Pis for monitoring and control of the buildings. The campus meter is also monitored from the node for the building where the campus meter is located.

2) BESS
The BESS at Scott Park is a Johnson Controls lithium-ion system with a 130 kWh capacity. It internally consists of three components: a partial MESA-Storage-compatible battery management system (BMS), a four-quadrant bidirectional 125 kVA inverter, and a meter. The BMS and meter are connected by Ethernet to an isolated Ethernet switch within the BESS container, while the inverter is a serial device connected to a Modbus TCP-RTU gateway which is in turn connected to the switch. A VOLTTRON node on a Raspberry Pi in the adjoining building is connected by Ethernet to the switch in the BESS.

3) PV
The Scott Park PV array is a 980 kW behind-the-meter First Solar CdTe system comprised of two independently metered sub-arrays separated by 200 meters -a 620 kW ''Office'' field with three inverters and a 360 kW ''Ballpark'' field with two inverters. At each field, there are two Modbus RTU serial networks, one comprising the inverters and the other containing the meters (and at the Office Field, the weather station). In their original configuration, these devices were connected to serial ports on a data logger monitored by a SCADA provider through an LTE modem at each field. Communication between the fields and the nearest campus building was established by adding fiber optic cables and Ethernet switches to create an isolated LAN, as shown in Fig. 3. The Modbus RTU networks were converted to Modbus TCP using RTU-TCP gateways to allow simultaneous access VOLUME 10, 2023 for both the data loggers and a VOLTTRON system on a Raspberry Pi in the adjoining building.

B. COMMUNICATION LAYER
Communication between assets on the campus is implemented using the VOLTTRON platform, as illustrated in Fig. 4(a). The VOLTTRON network consists of ten nodes located within the university's privately-addressed facilities IP network. Nine of these are installed on Raspberry Pis: six for the buildings, one each for the BESS and PV array, and one bastion node providing access to external resources beyond the private facilities network e.g. weather forecasts, data from electricity markets and, hypothetically, neighboring smart grid deployments. The tenth, installed on a laptop computer, serves as a central hub for the network and houses a historian agent which collects data from other nodes into the database.
The core of the VOLTTRON system is a message bus which facilitates communication within and between nodes using publish/subscribe and remote procedure call (RPC) methods. Between nodes, this communication is secured using the VOLTTRON Interconnect Protocol (VIP) using elliptic-curve cryptography [38]. VOLTTRON also provides authentication and authorization to secure against unauthorized access to system data and unauthorized use of control functions [37]. The private address space and university firewall provide a secondary layer of security.
Data points on devices, which at Scott Park use either the BACnet or Modbus protocols, are polled by VOLTTRON using a driver framework which abstracts away the details of this communication, freeing applications in the platform from the need for knowledge of the protocols for each device. Data is published on the message bus in a common format where it is visible in real-time, by subscription, to any other VOLTTRON agent, including the database historian and userinstalled control applications. Devices are actuated by sending a remote procedure call (RPC) to a special actuator agent which uses the driver framework to write to the device in its native protocol.
Further details of VOLTTRON integration for the various assets are as follows:

1) VOLTTRON INTEGRATION WITH BUILDINGS
All communication with equipment managed by the university building automation system uses the BACnet protocol. Architecturally, VOLTTRON occupies a similar place in the BAS hierarchy to Insight and, were the field panels native BACnet devices, could bypass Insight entirely. Because this older system uses the proprietary P2 protocol, however, and upgrading panels would have been cost prohibitive, integration is achieved by using an Insight module to map the underlying devices to BACnet. Communication from VOLTTRON to building devices is illustrated in Fig. 4(b). In total, 8,536 data and control points are mapped. Through these control points, VOLTTRON controls and monitors the devices in the BAS by writing at a higher BACnet priority than Insight. In the event that there is a need for the facilities team to override the decisions made through VOLTTRON, Insight operators can retake control by using a still higher priority [39].
It should be noted that VOLTTRON does not, in this setup, replace the building controls. Rather it provides an overlay and is used to modify the behavior of existing controllers by writing to setpoints. The existing PID controllers are then able to manage the low-level control of the devices while higher-level control algorithms are tested. Were it advantageous to directly implement lower-level controls, the PID controllers could also be bypassed by writing the registers used internally by the existing systems to actuate the devices themselves. Alternately, integration of third-party control algorithms might be achieved through a similar mechanism to what is used for interaction with PID controls.

2) VOLTTRON INTEGRATION WITH BESS
The BESS VOLTTRON node has two network interfaces, as shown in Fig. 4(c). The built-in Ethernet port is connected to the facilities private network and is used for communication with other VOLTTRON nodes while a second, added using a USB-Ethernet converter, is connected to the internal BESS LAN. In this manner, the devices within the BESS are not exposed directly to the university networks as all communication with the internal network is mediated by VOLTTRON. The driver framework communicates with the BMS, inverter, and meter using Modbus TCP.
For the variability mitigation experiments demonstrated in Section IV-A, speed is crucial to the performance of the algorithm. The maximum speed at which control can be achieved is dependent on three things: the speed at which the VOLTTRON driver framework can poll the devices configured for it, the speed with which the physical device can respond, and that of the bus on which the device is located. Experience has shown the most common limiting factor to be collision avoidance within the serial bus. In the development of the BESS controller, it was found that the Modbus devices within the BESS could be reliably polled at speeds at one second intervals. While this was the polling speed eventually used in the experiment, some modifications to the VOLTTRON ModbusTK driver were tested to ensure faster polling speeds would be possible if needed. The Modbus requests consist of a start register and the number of registers from the starting point to read. The device will then respond with all registers within this range. The VOLTTRON driver implementation did not have any specification for skipping registers, and so if several non-sequential registers were requested, multiple request and response frames would be sent. The tested modifications allow specification of ''pad'' registers. When all missing registers in a sequence were configured as pad registers, a single request would be made and the device would respond with the minimum number of frames necessary to include all registers in the range. Pad registers would then be ignored by the VOLTTRON driver. As a large number of registers can be returned within a single frame, this allowed cutting the amount of traffic substantially on the Modbus serial busses, allowing for poll rates approaching 0.5 seconds, making the response-time of the devices the new limiting factor.
Top-level control of the BESS is managed by a VOLTTRON agent, written by the authors. The code for this agent, intended to be compatible with the MESA-Device standard, has been contributed to the Eclipse VOLTTRON project. The source code and documentation are available in the VOLTTRON PNNL Applications Repository [40]. Charge or discharge rates, along with an offset from unity power factor, may be set using RPC. In addition to handling control requests, this agent handles all equipment interaction, including monitoring faults and managing state of charge (SoC) limits. In the case of the Scott Park system, SoC is disallowed from passing below 10% or above 90% thresholds, though this is configurable as the optimal strategy for safely handling the SoC of a given BESS depends on the architecture of the battery.

3) VOLTTRON INTEGRATION WITH PV SYSTEM
As was described for the BESS, the PV VOLTTRON node polls the inverters, meters, and weather station using Modbus TCP over one Ethernet port from the isolated network at the PV array and publishes this data over a second port to the VOLTTRON message bus. This integration is illustrated in Fig. 4(d).

C. DATA MANAGEMENT AND MONITORING LAYER
The first step to manage energy is to monitor it. While building automation systems do not typically keep long-term trend data, the extensive historical data collected by the testbed can facilitate automatic fault detection and diagnostic algorithms and provide new insight to researchers and facilities staff on energy efficiency and building management.
The database used for the Scott Park Testbed is a CrateDB cluster installed on seven virtual machines. As shown in Fig. 4(a), several nodes face the facilities network and accept writes from the VOLTTRON historian. One read-only node faces the university's publicly addressed network to provide access to the data. Three additional nodes (not shown), connected only to an internal LAN within the cluster, provide more storage. One additional VM is dedicated to Grafana, a visualization tool for time series data. Grafana dashboards have been developed for each building load, the BESS, and PV data. As an example, the BESS panel shown in Fig. 5 visualizes real-time or historical data for the state of the BESS including SoC, power, and voltage.

D. CONTROL LAYER
In addition to the communication facilities discussed above, VOLTTRON provides a robust environment for development of agent-based applications. Applications are written for the environment by subclassing the VOLTTRON Agent class for access to the message bus, historical data, authorization, and scheduling facilities. Once VOLTTRON has been configured to communicate with devices, ''agent'' applications can be easily developed to interact with them over the message bus, allowing the author to focus on developing control algorithms without the need to manage the native protocols of each device. The message bus can also be used to communicate with other agents running on any connected VOLTTRON node.
The testbed provides functionality to enable the complete cycle of control development, from conception to deployment. Data collected from real-world sensors and online sources is available in real-time and recorded to a database, providing the input for exploration of the problem domain and for models in offline simulation. More realistic simulations can be performed within the testbed environment itself, using VOLTTRON's co-simulation framework or through the use of custom agents as digital-twin models of the realworld devices to be controlled. Integration testing on the real network hosts of the testbed provides assurance that the both the algorithm and the necessary code for ingesting data and signaling is behaving as expected before actuating the controls on physical devices in a live test. Once the algorithm is deemed to be sufficiently mature, real-world testing can then be performed by allowing the algorithm to actuate the real-world equipment. Finally, if the algorithm is successful, it may remain deployed for its intended purpose, should it be deemed beneficial to the campus to continue its use.
As part of the Scott Park Testbed, numerous agents have been implemented to perform real-time data analysis, calculation of aggregate metrics, polling data from Internet sources, and control. These include the BESS controller agent, discussed in Section III-B.2, and those used for PV variability mitigation, discussed in Section IV-A. It is also possible to leverage the work of other developers who have made agents available to the open source VOLTTRON community. An example of this is the Intelligent Load Control agent [41], deployed here by the authors for peak load control and load shaping experiments in buildings. These experiments are discussed in Sections IV-B and IV-C.

IV. CONTROL EXPERIMENTS
The Scott Park Testbed has been used to validate control strategies for mitigating the effect of the variability and uncertainty of renewable generation by effectively managing DERs. This section presents the implementation of several control experiments with the buildings' flexible load and a battery. Section IV-A discusses the mitigation of PV power variability with the BESS using a novel algorithm on the Scott Park Testbed. As renewable generation depends on meteorological conditions, it can also lead to uncertainty in power generation and therefore affect the power system's stability. Due to the advancement of meteorological prediction techniques, however, this imbalance can be predicted beforehand [42], [43] and its effect mitigated by load management or load shaping [11]. In section IV-B, peak demand is managed by modifying load power using the Intelligent Load Control (ILC) algorithm developed at PNNL. Section IV-C illustrates a different application for the ILC agent, wherein it is used to shape a campus building's load profile by following a pre-determined schedule in preparation for ongoing work with transactive energy markets. In addition to the experimental outputs, this paper presents how VOLTTRON agents operate in different layers of the control-centric framework.

A. MITIGATION OF PV VARIABILITY WITH BESS
Distribution-side PV generation is growing rapidly due to improvements in technology and production costs. Intermittent cloud cover, however, causes variability in output power, which can hamper grid stability [44]. This PV power variability can be mitigated by using a BESS at the common coupling node with the grid.
A distributed system for PV mitigation can be constructed as shown in (1), where P Campus−Grid is power flow from the distribution feeder to the campus. P PV is power generated by PV. P BESS is power flowing into the BESS.
The authors of this paper have previously developed and discussed [4] a mitigation algorithm which uses adaptive moving average control (AMAC) and adaptive state of charge (ASC) strategies to mitigate the PV variability with a better trade-off than existing techniques between the battery capacity requirement and the degree of the smoothness. AMAC dynamically changes the window size of a low-pass moving average filter in accordance with real-time variability metrics, reducing battery capacity requirements by scaling the response to the current level of variability. ASC uses feedback to adaptively modify target outputs to maintain the SoC close to its center point. This novel strategy was first developed in simulation and then validated experimentally on the testbed. Formulation and analysis of both AMAC and ASC are detailed in [4].
The herein presented work focuses on implementing the algorithm presented in [4] on the proposed testbed, and elaborates on how this framework helps to execute the algorithm seamlessly. The unpredictability and fast timescale of PV power fluctuations demand a control system that quickly and accurately runs mitigation algorithms and promptly actuates the BESS in response to variability. VOLTTRON enables secure and fast communication between these devices and agents in the control layer. Agents in the control layers perform their operation distributively and communicate via VOLTTRON's message bus. In addition to the BESS agent discussed in Section III-B.2, this process involves two agents, each managing a separate part of the process:

1) AMAC-AGENT
This agent subscribes to real-time PV power data on the VOLTTRON message bus and calculates its variability, σ PV ,W , using a moving standard deviation as shown in (2), where W is a window size of the moving standard deviation. The AMAC algorithm is implemented by using this variability metric to calculate an adaptive window size T(t), given in (3), for the adaptive moving average (P smooth AMA ) (4). The parameters T max , σ min , σ max , and a are, respectively, the maximum allowed window size, the minimum variability, the maximum variability and a damping factor. The selection criteria of these parameters for specific power rating systems are briefly explained in [4], and they are defined in an agent configuration file so that users can modify them quickly for different systems. The variability metric, adaptive window size, and moving average are all published to the VOLTTRON message bus for use by other agents and archived in the database for real-time analysis.

2) MITIGATION-AGENT
This agent subscribes to state information published by the actuation device as well as the adaptive moving average and variability metrics published by the AMAC-agent. It uses this data to manage control decisions for the BESS. It uses the instantaneous value of SoC published by the BESS and the AMAC signal published by the AMAC-agent to generate an ASC correction signal. Then, the proposed smoothing control, a combination of ASC and AMAC, is calculated and used to actuate the BESS.
To keep the SoC near its reference value (SoC ref ) and allow the use of less battery capacity for the same degree of smoothing control, the SoC needs to be adjusted with SoC droop control (SoC Droop (t)) as shown in (5), where SoC max is maximum allowable SoC, and SoC(t) is instantaneous SoC.
Using the maximum power (P BESS max ) for SoC droop control may inject a small amount of variability. This additional variability can be mitigated by using an acceleration parameter (σ adp ), shown in 6, to linearly modify the SoC droop control parameter.
Henceforth, adaptive SoC control (ASC) correction signal (P asc ), determined from the SoC droop control parameter (SoC Droop ) and acceleration parameter σ adp , as shown in (7). P asc (t + dt) = P BESS max · min(σ adp (t) · SoC Droop (t), 1) (7) Finally, the power for smoothing control (P smooth ) is calculated by (8) and the operation signal for BESS is calculated by (9) with a switching element such that the battery only operates when the adaptive moving window T (t) is positive. The BESS's operation signal is used to actuate the BESS using an RPC command to the BESS agent. P smooth (t) = P asc (t) + P smooth AMA (t) (8) Separating actuation from the calculation of need into distinct agents allows for development of cooperative or marketbased strategies wherein multiple devices manage a larger need than a single device. Ongoing work uses building HVAC systems with the actuation mechanism demonstrated below in Section IV-C to manage larger and lower frequency variation. Alternately, multiple BESS units may be used for scale. The modularity of the testbed readily facilitates this iteration and reconfiguration to test multiple variations of the control strategies under development.
The testbed allows reliable, distributive, sub-second communication, making it well-suited for validation of this type of control strategy. By collaborative use of these agents, a field test was conducted. Experimental results are depicted VOLUME 10, 2023 in Fig. 6. Fig. 6(a) shows an overlay of PV power with the results of simulated and experimental smoothing. The root-mean-square deviation between the simulated and experimental values is a mere 7.08 kW. This shows a good agreement between the simulated and experimental results. As shown in Fig. 6(b), the SoC is maintained close to the reference SoC of 50%, moving bidirectionally above and below the reference. Traditional moving average methods tend to rise, in an unidirectional manner, well above the reference in the early part of the day, leading to a larger capacity requirement for the battery [4].

B. PEAK DEMAND REDUCTION
Demand charges can be a significant portion of utility bills for large electric customers, making management of demand peaks important for reducing energy costs. This is also important for distribution utilities as the system must provide sufficient capacity to support peak demand. Duty-cycling of loads has traditionally been used to manage peak demand. Duty-cycling, however, does not prioritize those loads with the greatest flexibility or highest occupancy and may cause thermal discomfort to occupants. Control strategies which account for priority and comfort may offer greater value.
The ILC algorithm manages loads using an analytic hierarchy process (AHP) by dynamically prioritizing the available loads for curtailment. Detailed formulation and use cases of the ILC are demonstrated in [45] and [41]. ILC has three basic elements: a goal, criteria, and alternatives. The goal can be to mitigate peak consumption or to maintain an energy budget. The criteria specify metrics that can be used to rank alternative loads for curtailment. These criteria can be used, for instance, to limit impacts to occupant comfort. ILC's ability to modulate load based on user-defined criteria and its prioritization capability makes it more attractive than other peak demand reduction techniques.
In ILC, a pairwise comparison of criteria is first conducted to qualitatively determine the relative importance of each criterion. The pairwise comparison matrix (A = a nxn ) of n criteria can be determined by comparing row (a i ) with column (a j ). If the i th criterion is ''x'' times more important than the j th criterion then a ij = x and a ji = 1/x. However, when i = j then a ij | i=j = 1. The principal eigenvector (A P ) determines the order of importance among criteria and can be calculated by averaging across the rows of normalized matrix A as shown in (10). The normalization of matrix A indicates the weight of each criterion and is calculated by multiplying the inverse of each column summation.
Most importantly, the pairwise comparison between criteria must be consistent. For example: x1, x2, and x3 are established as numerical rankings for three different criteria; if x1 > x2 and x2 > x3, then x3 should be less than x1. If x3 > x1, then the pairwise criteria is inconsistent. Therefore, a consistency test needs to be performed before finalizing the eigenvector by measuring the consistency ratio (CR) as shown in (11), where, l max is the maximum eigenvalue of matrix A and can be calculated using (12). The appropriate value of the random index RI with respect to n is provided in a table presented in [46].
The alternative/load matrix, (B = b nxm ), can be created using the importance of each of n loads relative to each of m criteria. Finally, a priority vector can be calculated as the scalar product of the transpose of the principle eigenvector (A T P ) and the normalized alternative matrix B n as shown in (13), where B n is calculated by multiplying the inverse of each column summation of matrix B.
To implement ILC, it is necessary to find a suitable shortterm load forecasting method to anticipate what the building demand would be for the next control interval. The exponential moving average (EMA) is a suitable time series approach as it gives more weight to the most recent value. An agent implementing the algorithm is available in the VOLTTRON PNNL Applications Repository [40]. This ILC agent was tested at Scott Park, using a maximum power threshold as the goal. When the EMA of building power exceeds the threshold, ILC evaluates its criteria using AHP to choose loads for curtailment to lower the building power below the threshold. The criteria used were recent curtailment count, historical zone temperature, deviation from zone temperature setpoint, zone airflow ratio, rated device power, and room type. The last is a categorical setting with numerical priorities given to zones based on their occupancy level and typical use. The other criteria are evaluated periodically by the agent from device data. For modifying power, the ILC agent alters the temperature setpoints of zones via the BAS.
Results from experimental validation of ILC are shown in Fig. 7. Figs. 7(a) and 7(b) show the exponential moving average of building power without and with ILC, respectively, and the maximum allowable peak demand threshold (set at 50 kW). When the total demand is under the threshold, there is no need for correction. Once the threshold is exceeded, ILC activates and chooses zones to curtail as shown in Fig. 7(c). Out of eighteen zones in the building, five were curtailed on the first attempt. For a single zone, curtailed for part of the active period, Fig. 7(d) shows the changes in setpoint and temperature. Without ILC, the temperature would be expected to closely follow the nominal setpoint at 74 • F. The shaded active regions have a shorter duration than for the building as a whole (as in Fig. 7(b)) because ILC is choosing zones as needed, based on their priority, to meet its objective. As one zone approaches its configured 2 • F constraint for temperature deviation, another zone is chosen, if needed, to meet the goal. Fig. 7(e) shows the corresponding changes in the PID loop and airflow in the same zone. The airflow change causes the AHU to alter its supply fan speed to maintain the static pressure of the supply duct, modifying the electrical load of the fan and thus the power consumption of the building.

C. LOAD SHAPING ON A SCHEDULE
There are scenarios where more complex shaping of loads may be beneficial for managing uncertainty. For instance, augmenting power when energy costs are low may reduce the need for curtailment later by storing energy in the air mass of a building. Transactive energy markets -an area of current work for the authors -may require compliance with schedules which have been bid into a market. The ILC agent used in Section IV-B has the capability to receive new targets by subscription on the message bus and is able to prioritize zones for the augmentation of power using a parallel set of criteria. These capabilities were tapped to demonstrate adherence to a variable, 10-minute to hourly building power schedule as shown in Fig. 8.
In this experiment, ILC was configured for bidirectional control and given threshold constraints of 3 kW above or below its target. A dynamic target goal was then published to the message bus for ILC to follow. As shown in the Fig. 8(a), whenever the building power hits the upper threshold, ILC curtails the power consumption of building loads. Similarly, whenever the building power is on the verge of falling below the lower threshold, ILC augments the power consumption of the building. Fig. 8(b) shows the change in temperature and temperature setpoint of one zone during the experiment. The temperature setpoint changes 2 degrees upwards or downward as necessary, and building power changes accordingly.
Utilizing ILC to bidirectionally shape the load of buildings grants the Scott Park testbed the means to test multiple scenarios for cooperative power and energy management. These include extending the work discussed in section IV-A to include campus buildings for mitigation of PV variability as well as for upcoming transactive energy experiments.

V. FUTURE WORK
Adaptive moving average control and intelligent load control methods are, of course, very different control algorithms for different problem domains, and they have consequently different time horizons. ILC does not operate at a highfrequency, at least in the context of building power, since the important metric for demand charges is typically the peak power over a 15-minute average. Having both subsecond and lower resolution control thus broadens the scope for implementing and testing various control methodologies for different problem definitions. While sub-second intervals were not found to be necessary for the variability mitigation experiment, one rationale for validating the capability, was to show the testbed would be appropriate for addressing use cases such as frequency control, which would also need to operate on very high-frequency time-scales.
The testbed is not limited, either, to running a single algorithm at a time. In fact, an area of future work is expected to be pairing of a fast algorithm such as the PV mitigation using the BESS with a slower-rate control of the same metric using buildings. This would allow the flexibility of the buildings to be used to further smooth the power at the utility interface by addressing the lower-frequency movements in power which remain after the BESS has smoothed out instantaneous fluctuations.
The experiments which have been performed on the testbed to date have not made use of occupant participation. As a result, the effects of energy management constitute, in economic terms, an externality, since occupants experience effects of decision-making without a direct mechanism for feedback. An interesting category of future work would include experiments that seek to internalize occupant feedback in decision-making process. For instance, organizational units might be given a budget for energy consumption and allowed to provide preferences in a transactive energy market. This would also facilitate study of the microeconomics and socio-psychological factors affecting energy consumption. Currently, market-based transactive energy strategies between actors within the electrical distribution system are being developed at the lab [47], [48].

VI. CONCLUSION
Given the growing adoption of renewable generation, it is necessary to develop control strategies for mitigating variability and uncertainty and to validate them in a real-world setting. This paper presents a control-centric framework and a living laboratory which has been implemented by integrating the VOLTTRON platform with the DERs of the University of Toledo's Scott Park campus. The testbed uses VOLTTRON to provide a unified messaging layer and abstracts away protocol specific aspects of communicating with equipment. Control Agents can then be written to prototype and test novel control strategies. Three control experiments have been illustrated to demonstrate the capabilities and flexibility of the testbed: PV variability mitigation with a BESS, peak demand management, and load shaping, with the latter two both using Intelligent Load Control. The demonstrated testbed, to the authors' knowledge, is the first of its kind to explore real world experimentation for mitigating variability and uncertainty in various time scales.
The system's flexibility facilitates an array of possible control strategies. By utilizing a real-world testbed, these strategies can be evaluated for effectiveness and practicality. Finally, the system's modern components and architecture enable the group to test relevant and promising technologies that will ultimately increase the adoption of DER while enhancing energy efficiency and the reliability of the electric grid.
SRINIVAS KATIPAMULA received the M.S. and Ph.D. degrees in mechanical engineering from Texas A&M University. He is currently working as a Staff Scientist at the Pacific Northwest National Laboratory (PNNL), Richland, WA, USA. He is a fellow of the American Society of Mechanical Engineers and American Society of Heating, Refrigeration, and Air Conditioning Engineers. For more than 25 years, his career has focused on research to improve the operating efficiency of commercial buildings and building-grid integration. Before joining PNNL, he led the analytics group at Enron Energy Services. Previously, he also worked at the Energy Systems Laboratory, Texas A&M University. He has extensive technical experience in building-grid integration, evaluation of advanced building systems design concepts, development of automated fault detection and diagnostic techniques, self-correcting controls, building and energy system simulations, and analysis and evaluation of new energyefficient technologies. He studies photovoltaic (PV) materials, devices, and systems in support of advancing this critical energy technology for both terrestrial and space applications. His research methods include ultrafast and steady-state optical spectroscopies, solutionprocessed and nanoscale engineered materials for PV applications, device modeling, and PV system modeling with energy production forecasting. After 14 enjoyable and productive years at the National Renewable Energy Laboratory (NREL), CO, USA, he joined UToledo in 2008 to focus on the application of advanced materials and characterization to commercially relevant solar energy technologies. Currently, he serves as the P.I. on two AFRL-funded research and development projects focused on the development of advanced thin-film PV technologies for high-altitude and space applications. Additional interests include bringing the benefits of clean solar electricity to the underserved communities in and around Toledo, OH, by working with the charitable non-profit organization, glass city community solar.