Evaluation of Data-Enhanced Hierarchical Control for Distribution Feeders With High PV Penetration

Reliable and resilient grid operations with high penetration levels of distributed energy resources (DERs) can be achieved with improved situational awareness and seamless integration of DERs with utility enterprise controls. This paper presents the details of the development of a data-enhanced hierarchical control (DEHC) architecture and the results of its evaluation. The DEHC is a hybrid control framework that enables the efficient, reliable, and secure operation of distribution grids with extremely high penetrations of solar photovoltaic (PV) generation by seamlessly integrating centralized utility controls, distributed controls for DERs, and autonomous grid-edge controls. In the DEHC architecture, the advanced distribution management system (ADMS) controls the legacy devices (such as load tap changers and capacitor banks), the PV smart inverters are dispatched by real-time optimal power flow, and the grid-edge devices regulate local voltages in coordination with each other. The DEHC is demonstrated using a commercial ADMS platform, real utility distribution feeder models, and grid-edge devices. The performance of the DEHC architecture is evaluated using simulations and hardware-in-the-loop experiments with voltage regulation as the control objective. The results show that the DEHC enables high penetration levels of PV in distribution feeders by effectively managing system voltages through the synergistic operation of ADMS, distributed PV smart inverter controls, and secondary-level grid-edge device control.


I. INTRODUCTION
With a solar photovoltaic (PV) generation capacity of nearly 400 GW worldwide and approximately 60 GW in the United States [1], high PV penetration levels exist in many distribution feeders today. The high PV levels can cause power quality issues, including voltage rise above ANSI limits [2], and high-voltage variability and flicker caused by the variability of the irradiance. These issues necessitate the development of new control frameworks The associate editor coordinating the review of this manuscript and approving it for publication was Ahmed Aboushady . that comprehensively address the challenges that exist in distribution feeders with high penetrations of PV.
The voltage issues in high-PV distribution grids can be mitigated by PV smart inverter functionality. Current research focuses on using autonomous Volt-VAR and Volt-Watt control strategies [3]- [5]. In this approach, a predefined Volt-VAR (or Volt-Watt) curve will be provided for each inverter, and the inverter will measure its local voltage to determine its reactive (or active) power output based on the curve. As demonstrated in [3], however, a significant drawback of the local autonomous Volt-VAR control is that the performance highly depends on the design of the curve. These curves might not be optimal for all operating conditions on the grid. These strategies do not accommodate network constraints or allow operators to leverage the flexibility from PV inverters for real-time operations. Aggregated voltage control methods using feedback-based real-time optimization overcome this issue. These methods use realtime measurements and network information to optimize the active and reactive power set points for PV inverters. The existing approaches employ voltage measurements [6], [7], manifold-based algorithms [8]- [10], and distributed control strategies [11]- [13] to enforce voltage regulation in distribution networks. Further, the online algorithms [14]- [16] can model a dynamic operational environment using timevarying AC optimal power flow (ACOPF) formulations in distribution grids. The hierarchical control approaches proposed in [17]- [19] use small test systems and do not study the coordination with the existing enterprise controls. Furthermore, several works reported multi-timescale control approaches to manage the legacy devices and dispatch DERs. In [20], a multi-timescale control strategy considering the spatial-temporal correlations of DERs is proposed to control the legacy devices and DERs including PV and battery storage. A dual-stage coordinated control approach for voltage regulation and congestion management is proposed in [21] for distribution networks with PV and electric vehicles. The first and second stages are designed to correct the long-term (using legacy devices) and short-term voltages fluctuations (using DER inverters), respectively. Reference [22] and [23] proposed three-level control strategies to minimize voltage problems by coordinating legacy device operations and PV smart inverter functionality. Model predictive control is used in [24] and [25] for the multi-timescale control in which the legacy devices are operated on slow timescale and the DER inverters are dispatched on the fast timescale for the VVO. In addition to PV, [25] also considers battery energy storage for the dispatch on the fast timescale. All these recent works on the multi-timescale control used small balanced test systems such as 33-bus, 38-bus, and 119-bus for the studies. However, the real distribution networks are typically larger in size and almost invariably unbalanced. As such, the existing works in literature are more suitable for the development of a proof-of-concept. The focus of this paper is to evaluate the DEHC architecture from a practical application standpoint. In this study, we used commercial ADMS, detailed distribution network models developed based on the data from the utility partner, and proprietary grid edge device models from the vendor in the implementation of the DEHC to ensure that the study is more realistic. We further validated the performance of the DEHC via hardware-in-the-loop experiments. Given the increasing penetrations of solar PV, a novel grid operations architecture that seamlessly integrates these aggregated voltage control methods with the existing enterprise controls is essential.
While we explore the integration of PV systems with grid operations, we must be cognizant of the following trends that are shaping grid modernization in the utility industry. The first is increasing investments in advanced distribution management system (ADMS) solutions [26] and the ongoing development of advanced control methods to enable the efficient, reliable, and secure control of distributed energy resources (DERs). The ADMS is an integrated platform that combines the functionalities of distribution management systems, outage management systems, and supervisory control and data acquisition (SCADA) systems for optimized distribution grid operation. A common practice utilities follow in the ADMS deployment is to deploy and operate a subset of the functionalities that constitute the ADMS platform to avoid high initial investment. In such a situation, system integration can be a challenge if the utility decides to add a new functionality offered by a different vendor at a later date [26], [27] because of the incompatibilities between the existing systems and the new functionality.
A second trend is the utility desire for additional levers in the form of grid-edge controls [28]- [31] to supplement traditional voltage control devices, such as capacitor banks. These devices are deployed on secondary-side networks to regulate the customer voltages within ANSI limits [28], [29]. However, the system integration challenges caused by the incompatibilities mentioned earlier hinder the widespread deployment of ADMS and other advanced controls [26], [27]; thus, a control architecture is needed that seamlessly integrates the fast-acting distributed control with the centralized control to enable effective decision making at timescales that match the dynamics of variable generation. To this end, we present a unique and innovative data-enhanced hierarchical control (DEHC) architecture that enables the integration of high PV penetrations and the emerging gridedge controls with the existing utility enterprise controls. In this architecture, the ADMS control decisions are enhanced by the measurement data obtained from the PV smart inverters and the grid-edge devices through grid-edge management system, hence the name DEHC. The DEHC architecture is shown in Fig. 1.
The DEHC architecture aims to achieve optimal and reliable operation of distribution systems by enabling a hybrid control approach. In this approach, the centralized control layer, which comprises the ADMS controlling the substation and legacy assets, will be complemented by (a) distributed PV smart inverter controls that are realized by real-time optimal power flow (RTOPF) algorithms [14], [32], [33] implemented on the smart PV inverters and on a coordinator -which could reside on the ADMS as shown in Fig. 1 or on a standalone system -and (b) autonomous control of the grid-edge devices. A commercial ADMS platform [34] is used for the centralized control layer in this work. Edge-ofnetwork grid optimization (ENGO) devices [30], [35], [36], which function as distributed controllable Volt-VAR resources on the low-voltage secondary network, are used for the grid-edge devices. The control set points for these devices are issued periodically by the Grid-Edge Management System (GEMS) head-end server. Compared to prior work that developed Volt-VAR/Volt-Watt control strategies based on local measurements [37]- [39], the RTOPF algorithm used in this work has the advantage of ensuring system-level optimality while leveraging fast response offered by the power electronics interfaces of the PV smart inverters. The proposed DEHC uses ADMS, RTOPF-based distributed energy resource management system (DERMS), and grid-edge technologies available as on date to the utility industry. These technologies are superior compared to the conventional DMS in terms of achieving improved scalability, controllability, observability, and hosting capacity.
The main contributions of this work are: 1. Implementation of the DEHC architecture by integrating the ADMS, RTOPF, and grid-edge device controls. The work in [32] is extended by integrating a real ADMS platform for the software simulations and grid-edge device hardware for the hardware-in-the-loop (HIL) experiments. This implementation facilitates re-creation of a utility control center environment in the laboratory to evaluate the performance of the controls in the planning scenarios of interest with high PV penetrations that may not currently exist in the field. 2. Evaluation of the DEHC approach using real-world distribution feeders through simulations and HIL experiments using the ADMS test bed [40]. A co-simulation platform is applied using the Hierarchical Engine for Large-scale Infrastructure Co-Simulation (HELICS) framework [41] to perform co-simulations and enable synchronous data exchange among the DEHC components for the evaluation.  3. Demonstration of the scalability of the DEHC approach on a large distribution system with more than 13,000 buses. Results show that the DEHC architecture enables the integration of very high penetration levels of solar generation up to 200% relative to minimum load in real-world distribution feeders without causing voltage issues. This is important for the practical implementation and deployment in the field. 4. Offer guidance for the distribution utilities in integrating highly distributed sensor measurements, especially those from the behind-the-meter PV systems, into the utility data systems.
In the remainder of this paper, Section II discusses the key features of the DEHC. Section III presents the distribution system modeling and the DEHC simulation details. Section V includes the simulation scenarios and results. Section VI discusses the DEHC framework evaluation using the HIL experiments, and Section VII provides the conclusions.

II. KEY FEATURES OF DATA-ENHANCED HIERARCHICAL CONTROL
The DEHC architecture enables a hybrid control approach, shown in Fig. 2, through systematic integration of three systems: 1) the ADMS, 2) grid-edge devices with a headend server, and 3) PV smart inverters and a coordinator with RTOPF controls. The functionalities of these systems are summarized in Table 1. The key features of the DEHC are: • ADMS-centered operation • Grid-edge voltage regulation • Distributed PV smart inverter control using RTOPF • Coordination of controls.

A. ADMS-CENTERED OPERATION
The ADMS is central to the DEHC architecture-it synergistically coordinates the operation of legacy assets, grid-edge devices, and PV smart inverters. This study uses the ADMS system developed by Schneider Electric [34]. The ADMS includes a full-scale network model populated from the geographic information system, customer information system with substations, and SCADA points were added to leverage advanced applications for network analysis and control. The real-time ADMS instance receives the measurements from various devices in the field via the SCADA system. The ADMS executes advanced model-based optimization using the built-in Volt-VAR optimization (VVO) application to find an optimal control strategy and issues commands to legacy field devices (tap changers and capacitors) and grid-edge devices.
The VVO is a model-based multi-objective constrained optimization procedure, with user-definable optimization criteria and constraints. In this work, the customer voltage improvement (voltage regulation) is selected as the control objective and the ANSI voltage limits are selected as the voltage constraints for the VVO. The VVO is configured to run every five minutes to issue the control set points for the legacy devices.
The legacy device statuses including the LTC tap position and capacitor on/off statuses are determined by the VVO module of Schneider Electric's ADMS. The VVO runs a proprietary multi-objective optimization considering the LTC/voltage regulator tap movement constraints. Note that the real distribution feeder network considered in this work does not have line voltage regulators. If the line voltage regulators are available, their tap positions are also determined by the ADMS VVO considering the corresponding tap movement constraints. Typically, 5 -20 tap changes are allowed in a day [42]. Five to six LTC tap changes are observed in the simulation results corresponding to one full day which are reasonable.

B. GRID-EDGE VOLTAGE REGULATION
The ENGO devices [43], referred to as grid-edge devices in this paper, provide increased flexibility in controlling the voltage profile in distribution feeders. These devices use power electronics-based, fast-acting, decentralized shunt-VAR technology for voltage regulation. Each device, which is connected to the secondary side of a pole-or padmounted service transformer, can inject 0 to 10 kvar reactive power and can regulate the voltage tightly (±0.5% within control range) at the service transformer. When sufficient numbers of these devices are placed correctly on the feeder, feeder-wide regulation can be achieved. The grid-edge devices are interfaced to the grid-edge management system (GEMS), referred to as head-end server in this paper, via a communications link to receive voltage reference signals. The GEMS communicates with the ADMS periodically to receive optimal voltage set point updates for the grid-edge devices. Although the grid-edge devices are autonomously controlled once a set point is dispatched, they can be provided with a new set point voltage via the grid-edge device head-end server at regular intervals (minutes to days, as scheduled).

C. DISTRIBUTED PV SMART INVERTER CONTROL
The RTOPF algorithm [32], developed based on the ACOPF model [14], is used to implement the distributed PV smart inverter control. The distributed control scheme [14] leverages the fast feedback and regulation capabilities of power electronics-based PV smart inverters to enable real-time control and ensure system-level optimality. The following presents the details of the RTOPF algorithm [14], [32].
Let N := {1, . . . , N } be the set of nodes in the distribution circuit and N PV ⊆ N be the set of nodes where PV inverters are connected. Consider that X t := {p t j , q t j , j ∈ N PV } is the vector of actual active power output, p t j , and reactive power output, q t j , from the j th PV inverter at time t. Let S j be the rating of the j th PV inverter. Then the RTOPF problem can be formulated as (1).
In this formulation, the objective (1a) is to minimize the active power curtailment and reactive power output from all the PV inverters. The constants w p and w q , such that w p w q , set the relative weights of the two terms in the objective function. The PV active power output at time t is limited to p t,max j by the available solar irradiance level at that time step, which is modeled by the constraint (1b). Additionally, the constraint (1c) ensures that the apparent power output remains within the inverter rating. Further, the PV inverters are assumed to operate in night mode, i.e., the inverters can generate and absorb reactive power even in the absence of active power output from the PV array. The node voltages in the distribution feeder should not exceed the ANSI limits, which are represented by the constraint (1d).
The linear power flow model proposed in [33] to represent three-phase unbalanced power flow is used to solve for the node voltages. Then the linear approximation of the complex node voltages (V ) and node voltage magnitudes (|V |) are expressed as linear functions of node power injections as: where p inj and q inj are the active power and reactive power injection vectors, respectively. A, B, C, and D are the power coefficient matrices for the linear power flow model; and k and l are constant vectors representing errors terms.
The RTOPF problem (1) is solved by the primal-dual gradient algorithm with voltage measurement as feedback [14], [32]. Let L t (X t ,μ t , µ t ) denote the Lagrangian function associated with the problem (1) at time t, whereμ t and µ t denote the Lagrangian multipliers for the upper and lower voltage limit constraints, respectively. Then the active and reactive power set points for the PV inverters are solved at the beginning of the next control time window (t + 1) as: where α x is the constant step size, and x L is the gradient. The Lagrangian multipliers are updated as: where αμ and α µ are the constant step sizes; and μ and µ are the gradients, which can be computed directly based on the voltage measurement feedback at last time step, v (t−1) . The gradient, x L, depends on x (t−1) ,μ (t−1) , and µ (t−1) , as well as the matrices A, B, C, and D.
Feedback-based real-time optimization using linearized power flow model [14], [16] is used in this work. We did not use any commercial optimization solvers such as CPLEX. The optimization is solved using gradient descent algorithm. Specifically, the equations (3) and (4) are solved iteratively.
The performance of the RTOPF depends on the hyperparameters such as weights in the objective function (w p , w q ) and the gradient step sizes (α x , αμ, α µ ). The weights w p and w q are selected such that w p w q so that the voltage regulation is ensured primarily by the reactive power compensation without relying heavily on the PV active power curtailment. In this work, these weights are selected as w p = 1 and w q = 0.001. The gradient step sizes impact the convergence performance of the RTOPF. Increasing the values of αμ and α µ accelerates the convergence but could result in oscillations and instability. On the other hand, decreasing these values result in slow convergence and poor performance. After parameter tuning, these values are selected as alpha αμ = 50 and α µ = 50. The gradient step size α x affects the settling time of the PV local controller. Its value is selected as 5 to achieve a reasonable settling time of 1 -2 minutes.

D. COORDINATION OF CONTROLS
The novelty of this work is the seamless integration of multiple voltage-regulation technologies, both at central and grid-edge levels, to achieve reliable system operation under high PV penetration levels. The existing approaches focus on proof-of-concepts using small test systems, limited control timescales and ignore coordination with the existing distribution management systems. To be applicable for the field deployment, We study the integrated operation of the multi-timescale controls including the existing enterprise controls on real large-scale distribution systems. The controls in the DEHC architecture operate in three timescales: slow, moderate, and fast. The coordination of different controls in these timescales is depicted in Fig. 3. The ADMS performs VVO every 5 minutes (the typical range is 5-60 minutes) and issues set points to the legacy devices in the slow timescale (T 0 , T 1 , T 2 , ..). The RTOPF solves the optimal power set points once every 5 seconds (the typical range is 1-60 seconds) and issues the set points to the PV inverters in the moderate timescale (t 0 , t 1 , t 2 , ..). When the local control is enabled in the PV smart inverters, their output changes take place in less than 1 second. The grid-edge devices respond to the local voltage changes based on their voltage regulation set point issued by the GEMS in the fast timescale, which is in the order of milliseconds. When VVO is performed, the ADMS computes the optimal voltage regulation set points for the grid-edge devices and communicates with the GEMS, which issues the set points to the devices installed in the field in the slow timescale.

III. DISTRIBUTION SYSTEM MODELING AND DEHC SIMULATION DETAILS A. DISTRIBUTION CIRCUIT DETAILS
A set of four distribution feeders supplied by a 30-MVA, 110-kV/13.2-kV substation transformer is modeled in OpenDSS based on the data received from the utility partner Xcel Energy. The feeders serve nearly 6,000 customers and have more than 13,000 buses in total. The topology of the system plotted using the GridPV tool [44] is shown in Fig. 4. In this system, the substation transformer is equipped with a load tap changer (LTC). Additionally, there are 13 switched capacitor banks, each rated for 1.2 MVAR, for voltage regulation and reactive power management. Based on the historical SCADA system data, the minimum load observed at the substation in the year 2018 is nearly 12 MW. To simulate a high PV penetration scenario, more than 3,000 distributed PV systems are added to the model with a peak generation capacity of 24 MW representing nearly 200% penetration level relative to the minimum load. Currently, there are 113 PVs in the field with 5% penetration level. The ZIP load model is used for the loads with the coefficients [Z, I, P] = [0.1, 0.8, 0.1]. Solar irradiance data recorded at NREL Flatirons Campus [45] at 1-minute resolution is used as input to all the PVs.

B. GRID-EDGE DEVICES
In addition to the legacy voltage regulation assets on the primary feeder, there are 144 ENGO devices present on the low-voltage secondary side to perform voltage regulation. The ENGO devices act as low-voltage static compensators to regulate the voltage at their terminals [43]. A set of 144 grid-edge devices as installed in the field is included in the OpenDSS model. The devices are deployed to improve the voltage profile using dynamic reactive power compensation [43]. Project partner Varentec supplied a proprietary model for OpenDSS in order to emulate the device functionality in the software simulations.

C. CO-SIMULATION WITH ADMS
The OpenDSS simulation of the feeder and ENGO devices is interfaced with the Schneider Electric ADMS using a co-simulation platform developed in Python software. This platform uses HELICS [41], an open-source co-simulation framework for electric power systems, as the core cosimulation engine. The co-simulation platform synchronizes the data exchange among the ADMS and OpenDSS simulation that includes the grid-edge devices. It streams the voltage and power measurements from OpenDSS to the ADMS as simulated SCADA and passes the optimal set points for the LTC, capacitor banks, and grid-edge devices from the ADMS to the OpenDSS model. These set points are applied to the simulated devices using the Python interface to the OpenDSS model.

IV. IMPLEMENTATION AND OPERATION OF DEHC ARCHITECTURE
The DEHC is implemented using a commercial ADMS, a prototype RTOPF-based DERMS, distribution simulator OpenDSS, real-time digital simulator OPAL-RT, HELICS, grid-edge management system (GEMS), grid-edge devices (ENGO), grid simulator, and real-time automation controller (RTAC) as shown in Fig. 5. The Schneider Electric's ADMS is a central controller to dispatch legacy assets such as LTC and capacitor banks. In our implementation, the ADMS runs the VVO every 5-minutes to compute the optimal set points for the LTC and capacitor banks. These set points consist of the capacitor bank on/off statuses and the voltage level to which the feeder head voltage should be regulated to using the LTC. The objective of the VVO is configured as voltage regulation, that is to maintain the bus voltages within the ANSI limits of 0.95 p.u. -1.05 p.u.
The distribution system power flow is solved in the OpenDSS simulator at each time step. The time step resolution is selected as 5-second. The communications interface between the ADMS and the OpenDSS is RTAC, which acts as a substation gateway. The ADMS and RTAC are configured as DNP Master and DNP slave, respectively. The RTAC receives the power flow measurement data from OpenDSS at each time step and communicates it with the ADMS as simulated SCADA. The simulated SCADA measurement data includes the feeder head real and reactive powers (P fh , Q fh ), line currents (I fh ), reactive power and voltages of the grid-edge devices (Q ge , V ge ), and the voltages of the PV systems (V pv ). These measurements are passed to the ADMS every 5-second interval so that the ADMS can assess the latest system state through state estimation. The ADMS communicates the LTC and capacitor bank set points (V LTC , S CB ) to the RTAC at each 5-minute interval which are then passed to OpenDSS where these set points are implemented. Since OpenDSS only supports UDP communication protocol and ADMS supports DNP3 protocol, the RTAC performs UDP to DNP3 conversion for the simulated SCADA data and DNP3 to UDP conversion for the legacy device set points.
The ADMS issues the voltage regulation set points to the grid-edge devices (V reg ) via the head-end server GEMS every 15-minutes. The grid-edge devices regulate their terminal voltages to the voltage regulation set points received from the GEMS by injecting the reactive power. This is how the grid-edge device communication works in the field. In our setup, three grid-device hardware units connected to phase A, B, and C receive the set points from ADMS in this manner. The rest of the grid-devices are simulated in OpenDSS. The set points for these devices are issued by the ADMS via RTAC. The PV smart inverter set points are issued by the RT-OPF which receives the target voltage operation limits (e.g., 0.95-1.05 p.u.) from the gateway as regulation targets and compute the active and reactive optimal power setpoints for all PV units. These set points are sent to the simulated PV systems in OpenDSS through HELICS.
The distribution system is primarily modeled in OpenDSS. A selected section of the system, referred to as subtree, is modeled in OPAL-RT as described in [46] for interfacing the three grid-device hardware units to replicate the real dynamics of the hardware devices in OPAL-RT. For these grid-edge device hardware units, the simulated terminal voltage is sent out through analog output to the grid VOLUME 10, 2022 simulator (Chroma) to reconstruct the physical voltage 240 V. The output voltage and current of each grid-edge device will be measured by a potential transducer and a current transducer and sent back to OPAL-RT via analog inputs to calculate the active and reactive power to control the current source and close the loop. The co-simulation is performed using the procedure described in Section III-C.

V. SIMULATION RESULTS
The grid integration of over 200 million DERs is expected by 2030 and the path forward for the distribution utilities is to rely on ADMS and/or DERMS solutions [47]. Research efforts are in progress to study the coordination of the ADMS and DERMS solutions, which represents the stateof-the-art, in achieving different control objectives such as peak load management, conservation voltage reduction, and VVO [32], [48]. In this paper, we evaluate the coordinated operation of ADMS and RTOPF-based DERMS through DEHC in achieving the voltage regulation using three simulation scenarios.
The scenarios summarized in Table 2 are simulated with voltage regulation as the control objective. In the baseline scenario, the legacy devices (LTC and capacitor banks) are in local control mode, the grid-edge devices are disabled, and the PV inverters inject power at unity power factor. In scenario S1, both the legacy and grid-edge devices are enabled and controlled by the ADMS. The feeder head power, voltage, and power measurements at all the legacy and gridedge devices are streamed from the OpenDSS model to the ADMS as simulated SCADA measurements at 5-second intervals. The ADMS runs VVO once every 5 minutes or when certain internal parameters exceed preset thresholds to compute optimal set points for the legacy and the grid-edge devices. The legacy set points include the desired voltage level at the LTC and capacitor bank on/off statuses. The PV inverters in OpenDSS are configured to follow the Volt-VAR control curve [49]. In S2, the same setup is used as for S1, except that the PV inverters follow the active and reactive power set points determined by the RTOPF.
In the local Volt-VAR control mode in S1, the PV smart inverters are assumed to follow the IEEE 1547 Volt-VAR curve [50] shown in Fig. 6. The curve settings are provided in Table 3. In this mode, the PV smart inverters inject or absorb reactive power based on the measured voltage. When the measured voltage is below 0.98 p.u., the reactive power is injected to boost the voltage. When the measured voltage is FIGURE 6. Volt-VAR curve [50] used in S1.  above 1.02 p.u., the reactive power is absorbed by the smart inverter to mitigate the voltage rise. There is also a deadband from 0.98 p.u. to 1.02 p.u. in which the PV smart inverters neither absorb nor inject the reactive power.

A. BASELINE RESULTS
The substation demand and the total PV generation in the baseline scenario are shown in Fig. 7. Because of very high solar generation, more than the peak demand, reverse power flow up to 10 MW is observed during the period between hours 10 and 16. The bus voltages plotted in Fig. 9 show that high voltages up to 1.08 p.u. are observed because of the high PV generation. Although the average voltage is close to 1.05 p.u during the peak generation period, low-voltage issues are not observed. Fig. 9 shows the voltage profile from the snapshot power flow simulation at the time step at which maximum voltage is observed. More than 400 customer locations experience voltages greater than 1.05 p.u.

B. SCENARIO S1 RESULTS
The VVO module in the ADMS supports multiple control objectives. For S1, the VVO objective is selected as the voltage profile improvement with the customer voltage constraints configured as ANSI voltage limits. The bus voltages from the simulation depicted in Fig. 10 show significant improvement in the voltage profile compared to    the baseline scenario. The LTC voltage regulation set points received from the ADMS and the resulting tap position status in OpenDSS compared to the baseline are shown in Fig. 11. With the fixed LTC voltage regulation set point in the baseline, the LTC tap position did not change. In S1, the ADMS performed VVO and reduced the LTC set point to limit the voltage rise caused by excessive PV generation.  As a result, the LTC tap position is reduced up to −6 and effectively mitigated the high voltages in the system. All the capacitor banks are kept on by ADMS VVO throughout the simulation period. The total PV generation in S1 compared to the baseline is shown in Fig. 12. Reactive power absorption is observed from hour 10 because of the voltage rise at the PV locations; however, because the PV systems are injecting active power up to the inverter rating near hour 12, reactive power absorption was not possible to reduce the voltages. Thus, voltages greater than 1.05 p.u. are observed near hour 12 in Fig. 9. There is no active power curtailment in S1. Overall, the improved voltage regulation in S1 is accomplished by the simultaneous application of hybrid controls-namely, the ADMS and local Volt-VAR control of the PV smart inverters.

C. SCENARIO S2 RESULTS
The system conditions in S2 in terms of VVO profile and control objective in the ADMS, the load, and the PV profiles are the same as in S1; however, the PV smart inverters are dispatched using the RTOPF algorithm. The bus voltages from the simulation, depicted in Fig. 13, show that all the bus voltages are within ANSI limits, which is not achieved in the other scenarios. The total PV generation in S2 compared to the baseline is shown in Fig. 14. A total energy curtailment of 16% is observed compared to the baseline to allow sufficient reactive power absorption to regulate the voltages within limits. The legacy device statuses, i.e., LTC tap position and the capacitor bank statuses, are the same as in S1.   The distribution of voltages in the QSTS simulation for all the scenarios are further compared in Fig. 15. The median voltage in baseline scenario is higher at 1.03 p.u. compared to the other two scenarios because of lack of reactive power absorption. The median voltages in S1 and S2 are 1.005 p.u. and 1.003 p.u., respectively which are nearly the same. The legacy device operation counts in each scenario are summarized in Table 4.
The results from the simulation scenarios are further quantified using the metrics total energy delivered by PV, PV energy curtailment, total energy delivered by the substation, and voltage exceedances. They are summarized in Table 5. The voltage magnitude of a bus being outside of the ANSI voltage range of 0.95 p.u. -1.05 p.u. is considered as the voltage exceedance. A bus can have multiple phases and each phase is considered as a node. The number of nodes exceeding the ANSI limits is multiplied by the duration of the exceedances. It is expressed in the units of nodehours. The total energy from PV is the same at 132.1 MWh in baseline and S1 scenarios since the PV active power curtailment is not allowed in these scenarios. This is lower in S2 at 111 MWh which corresponds to a PV energy curtailment of 16% compared to the baseline. The energy delivered by the substation is lower in S1 compared to the baseline despite the energy delivered by the PV is the same in these scenarios. This is due to the lower energy consumption by the voltage-dependent loads in S1 because of the lower bus voltages in this scenario compared to the baseline. The energy delivered by the substation is the highest in S2 due to the PV energy curtailment. The voltage exceedances are the highest at 3178 node-hours in baseline representing poor voltage regulation. The voltage exceedances are 9.7 node-hours in S1 which is significantly lower compared to the baseline. All the bus voltages are within the ANSI limits throughout the day in S2, i.e., there are no voltage exceedances.

VI. HIL EVALUATION
HIL evaluation is performed to further validate the feasibility and performance of the hybrid control approach enabled by the DEHC architecture shown in Fig. 2. The HIL setup is summarized here and further details on this setup can be found in [46]. The HIL setup used to run the experiments is shown in Fig. 16. This setup consists of the key elements: Schneider Electric ADMS (graphical user interface is shown); real-time simulator OPAL-RT, which simulates the part of the circuit where the ENGO hardware units are connected; ENGO hardware units communicating with the cloud-based head-end server (GEMS); grid simulator; and the laptop running the RTOPF algorithm, co-simulation software, and the OpenDSS power flow. An industrial gateway (a realtime automation controller) is used to support the protocol conversion for the data exchanged between the OpenDSS and the ADMS because the OpenDSS software does not support the DNP3 communications protocol required by the ADMS. This HIL platform provides realistic laboratory testing, including accurate modeling (legacy devices, gridedge devices, and PV) of the Xcel Energy distribution feeders, a real controller (ADMS), software controller RTOPF, hardware grid-edge devices, and standard communications protocols.
To validate the performance of the hybrid controls enabled by the DEHC architecture, a 3-hour testing period from 10:00-13:00 is selected for the HIL experiment of S2. Because this period of the day has high solar generation with fluctuations, the performance of the DEHC architecture in a challenging and realistic situation can be effectively validated. The results of this experiment are presented in Fig. 17 through Fig. 22. Fig. 17 shows that the bus voltages across the distribution feeders are regulated within the ANSI limits under high PV generation with the help of ADMS-centered and coordinated operation using the DEHC   architecture. Fig. 18 compares the changes in the ADMS set points and the corresponding LTC tap position changes in the S2 results collected from simulations and the HIL experiment. In both cases, the ADMS reduced the LTC voltage set point compared to the initial setting to reduce and regulate the voltage within the target limits. Consequently, the LTC tap position was reduced to −6 in the simulation and to −5 in the HIL experiment.
The voltage set point from the ADMS, the measured voltage, and feedback of one hardware ENGO are presented in Fig. 19 and Fig. 20, which show that the ADMS commands a low-voltage set point for the ENGOs to stop injecting reactive power. In Fig. 19, it is evident that the ENGO device received the updated voltage set point at 10:10, which is 10 minutes after the ADMS issued the set point. This delay is realistic, and it includes the delay between the ADMS and GEMS and between the GEMS and ENGO hardware. As observed in Fig. 20, the ENGO device operated as expected to inject reactive power when the voltage set point is higher than the measured terminal voltage.
The total PV output from the results collected in the HIL experiment of S2 is compared with the S2 simulation results   in the plots shown in Fig. 21 and Fig. 22. As observed in Fig. 21, the total PV active power with RTOPF control is nearly the same in the S2 HIL as that in the S2 simulation until 11:30 a.m. The active power curtailment is more in the HIL after 11:30 a.m. because the LTC tap position is higher in this case than in the S2 simulation. In both cases, the reactive power absorption is observed to keep the rising bus voltages within limits. The calculated total energy curtailment during the 3-hour period is 4.6% of that of the baseline when no curtailment is allowed. The plots in Fig. 22 show the response of one selected PV local controller, including the available power from solar irradiance and the active and reactive power set points. The local PV inverter controller responds correctly to high voltages by absorbing reactive power with minimal active power curtailment.  The HIL experimental results here show the response of the ADMS, legacy and grid-edge devices, and RTOPF in highvoltage scenarios, and they demonstrate the effectiveness of the ADMS centered and coordinated operation for voltage regulation. Based on the laboratory HIL evaluation, the utility can set up the same DEHC grid-automation system in the field to manage DERs, legacy devices, and grid-edge devices to improve the operation and performance of distribution systems with high PV penetration. This also shows how using HIL experiments help derisk the integration of new technologies proposed by the DEHC architecture in the field by integrating the systems with hardware in a laboratory.

VII. CONCLUSION AND FUTURE WORK
The increasing PV penetration levels in the power distribution networks are creating operational challenges in ensuring grid reliability and power quality. Voltage regulation is a key issue to be resolved to enable the grid integration of high levels of PV. Various advanced control solutions that perform the DER control at different levels of the distribution networks are available. It is essential that all these control solutions operate in a coordinated manner to achieve desired performance at the system level. In this paper, we presented a DEHC architecture, which integrates an ADMS, grid-edge device controls, and distributed PV smart inverter control via RTOPF. The developed framework is evaluated using real distribution feeders in software simulation and HIL experiments. The results show that the hybrid coordinated control approach enabled by the DEHC framework is effective in accomplishing the desired voltage regulation under high PV penetration levels up to 200% relative to the minimum load. The superior voltage regulation performance is realized by the ADMS-centered control of legacy assets, such as LTCs and capacitor banks, as well as grid-edge devices and the distributed control of PV smart inverters using the RTOPF algorithm. As part of the future work, we will conduct techno-economic analysis to quantify the investment costs to the utility in deploying the advanced controls and expected benefits. We will also expand the simulation scenarios to consider seasonal load and generation variations, various DER mixes, and explore additional control objectives such as conservation voltage reduction. The results from the future study will be disseminated to provide insights into the benefits and challenges that high levels of DERs present to the distribution grid operators.
JING WANG (Senior Member, IEEE) is currently a Senior Research Engineer at the National Renewable Energy Laboratory. Her research interests include microgrid control and simulation, distributed energy resources (DERs) integration, and control of DER inverters. She has expertise in power and controller hardware-in-the-loop (HIL) evaluation of advanced distribution management systems (ADMS), distributed energy resource management systems (DERMS) and DERs for grid automation and control, and DER integration studies. She leads HIL evaluations for multiple projects that use ADMS test bed.
SANTOSH VEDA (Senior Member, IEEE) received the B.E. degree in electrical and electronics engineering from Anna University, India, in 2008, and the M.S. and Ph.D. degrees in electrical engineering from Virginia Tech, USA, in 2010 and 2012, respectively. Currently, he is the Manager of the Grid Automation and Controls Group, Power System Engineering Center, National Renewable Energy Laboratory (NREL), Golden, CO, USA. At NREL, he leads a group of researchers working in the areas of advanced grid automation controls, microgrid controls development and validation, and integration of nontraditional DER systems. He also leads projects in the area of distribution grid modernization including deployment and performance evaluation of utility management systems like the advanced distribution management systems (ADMS) and distributed energy resources management systems (DERMS). He is also leading efforts on integration of electric vehicles and fuel cells. Prior to joining NREL, he was a Lead Researcher at GE's Global Research Center, where he developed advanced algorithms for grid control. His research interests include grid automation and control, advanced distribution management systems, distributed energy resources management systems, and microgrid control. His primary focus is solar power forecasting as a means for cost-effective integration of PV in the electric grid. His interests include PV system modeling, performance, monitoring, and reliability. Previously, he consulted for the World Bank Group. His solar career began at SunEdison, in 2008, as a New Technologies Analyst. He later became the Senior Manager of the Energy Analytics Group, where he developed algorithms for event detection, power forecasting, and inverter control, and generated analyses on power output variability and system reliability. In 2014, he became the Director of the Performance Analytics Group, where he was responsible for all performance reporting, analysis, and optimization. Prior to SunEdison, he worked for seven years in the semiconductor equipment and photonics industries as a Research and Development Manager and an Engineer.