Stochastic-Risk Based Approach for Microgrid Participation in Joint Active, Reactive, and Ancillary Services Markets Considering Demand Response

In the restructured power systems, renewable energy sources (RES) have been developed. Uncertainties of these generators reduce the reliability and stability of power systems. The frequency and voltage for the correct operation of the power systems must always be maintained within a nominal value. Ancillary services (AS), energy storage systems (ESS), and demand response programs (DRPs) can be effective solutions for mentioned problems. Microgrids (MG) can make an improvement in their profits and efficiency by participating in various markets. This paper provides an optimal scheduling for the simultaneous participation of MGs in coupled active, reactive power and AS markets (regulation, spinning reserve and non-spinning reserve) by considering ESS, DRPs, call for deploying AS, and the uncertainties of wind and solar productions. Capability diagrams; mathematical equations are used to model active and reactive power of generation units. Risk management in this paper is done by the conditional value at risk (CVaR) method and probability distribution functions (PDF) are used for modeling uncertainties of wind speed and solar radiation. The ERCOT (Electric Reliability Council of Texas) market is simulated with real world data.


I. INTRODUCTION
A. BACKGROUND D UE to low air pollution, high taxes for thermal generators, transmission line losses and etc., renewable products are an integral part of the restructured power systems [1], [2]. Extensive usage of renewable generators increases power systems uncertainties [3].
The difference between the rated voltage and the voltage of the power systems and the difference between the rated frequency and the frequency of the power systems causes many setbacks for generation units and customers (such as power systems blackout on September 28, 2003, in Sweden and Denmark) [4]. Therefore, they should be controlled in nominal values.
Wang et al. [5] illustrates the effects of the increasing capacity of RES in a sample power system. In addition, it analyses the positive effects of ESS and DRPs in the network. AS, DRPs, and ESS are suggested solutions to these problems that will be discussed in this paper. Various variables such as uncertainties of RES, risk management, and technical constraints effect the MGs optimal planning.

B. LITERATURE REVIEW
Hitherto, some articles have examined MGs participation in EM by considering effective points on the MGs decision; for instance, Ferruzzi et al. [6] considered the errors of predicting renewable generators and presented the dayahead (DA) optimal scheduling of MGs participation in the EM. Moreover, they used thermal power supply to reduce the cost and increase the reliability of the power systems. Shafiekhani et al. [7] presents a bidding strategy of virtual power plants in EM for maximizing their profit and reducing their emission by using a bilevel mathematical program with equilibrium constraints.
Consequently, some papers suggested effective policies for participating in AS markets. Majzoobi and Khodaei [3] presents different constraints for MGs capability in participation in AS markets (1-minute based frequency regulation, hourly ramping and 10-minute based load following) to reduce MGs cost and increase their profitability. Authors of [8] tried to coordinated different DERs (distributed energy recourses) such as WT, PV and ESS to maximize MG's profit. This coordinated DERs participate in energy, spinning reserve, and ramping markets; in addition, in the mentioned paper, hybrid stochastic/robust optimization approach was used to cover the volatile nature of photovoltaic and wind power.
The authors of [9] presented a joint clearing strategy of energy, regulation, reserve and ramping product from thermal generators and ESS. They tried to ensure that reserve delivery is guaranteed under all conditions. Providing inertial response and primary frequency control by RES. PV units will be able to provide the inertial response, by storing electrical energy in DC-link of inverters is the main idea of [10]. VOLUME 10, 2023 The authors of [11] tried to increase the profit of ESS by presenting a multi-stage stochastic bidding model. They considered energy, regulation, flexible ramping, DA and real time (RT) market. Authors of [12] suggested using a non-linear stochastic model for a high number of residential PV-battery systems in regulation up and down markets. In [13], ability of a distribution grid for taking part in frequency control market in grid-connected mode is discussed. In addition, it suggested control schemes for secure operation in islanded mode.
Wu et al. [14] develops an algorithm for managing MGs in EM and AS markets by a bilevel approach (lower level for a MG and upper level for a distribution system). Uncertainties are cover by a stochastic programming model. In [15], a bilevel stochastic programming approach is used to optimal planning of different sources in MGs. Moreover, Lognormal PDF is used to model the uncertainties of energy and spinning reserve prices. Khaloie et al. [16] uses a three-stage stochastic multi-objective to coordinate WT, thermal generator, and ESS bidding in energy and spinning reserve markets to maximize the generator's accepted revenue. In this paper, uncertainties of different parameters modeled using a scenario-based approach.
In [17], scenario generation method and uncertainties (of RES and loads) are considered for optimum scheduling of ESS in spinning reserve market. In [18], participation of renewable MG in DA reserve and energy markets is modeled by considering different uncertainties (load realizations, solar irradiance and wind speed) and CVaR. Authors of [19] considered different variables and uncertainties, though DRPs and reactive power are not considered.
Some of the reviewed papers added reactive power to MGs bidding strategy; for instance, in [20] arbitrage strategy of virtual power plants in coupled energy, reactive, and spinning reserve markets is modeled by considering power systems reliability and MGs profit. The first goal of [21] is to present a strategy to coordinate DER and traditional resources for providing a stable voltage and having safe and stable power system. In [22], a novel control scheme is presented to illustrate the undeniable effect of renewable AS in systems with high penetration of RES by using smart inverters of PV systems and limiting the voltage and frequency fluctuations.
In [23], a parametric cost-function is discussed for maximizing the benefit of DERs by participating in primary frequency response and voltage control markets. In [24], the participation of a VPP (Virtual Power Plant), with high penetration of PV, in voltage AS market is discussed, and a method for this purpose is suggested. An array of parameters, such as AS and reactive power, are presented in [25], though DRPs can be used to develop this reference.
DRPs are used to develop previous models for maximizing the profit of MGs in some papers. Wang et al. [26] suggests an optimal bidding model for the DER (ESS, EV, DG) aggregators taking part in regulation service, synchronized reserve, non-synchronized reserve and DRPs. It covers uncertainties by scenario generation methods. The authors of [27] presented a solution (MILP) for planning the participation of ESS and shifting DR (SDR) in balancing market. In [28], the authors discussed a two-level model for simultaneous participation in the energy and reserve markets considering the uncertainties of energy price and renewable generations, the possibility of call AS; moreover, the amount of participation in each market and the total profitability of the MG are the outputs of this model. Interruptible loads are briefly presented in this reference.
In [29], optimal planning for participation in EM is modeled considering the uncertainties of load, RES, and outage of power plants; incentive DRPs are used in this article. Vahid-Ghavidel et al. [30] suggests a hybrid stochastic-robust model for better operation of DR aggregators (time-based and incentive-based) considering the uncertainties of market price and participation rate of consumers. Samimi et al. [31] presents a stochastic framework for bidding in joint active and reactive using demand buyback program as DRP. In [32], the authors discussed incentive DRPs (load shifting and load curtailments) for DR aggregators in EM considering the uncertainty of prices, and explained technical and financial constraints of DRPs. In this article, CVaR approach is used for risk management.

C. CONTRIBUTIONS
As it is summarized in Table 1 and according to literature review, for increasing MG's profit and safety of power systems, authors suggest an optimal model for a MG ( fig.1) participating in EM consists of AS and reactive power markets. Particularly, an array of parameters should be considered to have an accurate model; however, all these parameters have not been considered in the literature to make the model complete for participating in the mentioned markets. Hence, presenting a novel and complete model for filling the mentioned gap is the main contribution of this paper. To this end, the proposed model considers an array of parameters such as participation in various markets (active, reactive, regulation up, regulation down, spinning reserve and non-spinning reserve), uncertainties of WT and PV, probability of call AS, and DRPs at the same time. Integrating three specific DRPs (load curtailment, load shifting and load recovery) including all practical constraints into the MG model is another novelty of this work. The CVaR approach is used for risk management. This strategy presents the cost model of generating units, probability of call reserve, and additional loss of production reactive power using capability diagrams.

D. PAPER ORGANIZATION
This paper organization is as follows: Section II illustrates the problem, presents voltage controlling, DRPs, the cost of different modes of power generation, uncertainties and risk management, and introduces objective function and constraints. Case study of real-world data is done in Section III. Section IV is conclusion section.

II. PROBLEM DESCRIPTION AND MATHEMATICAL MODELING
Power systems need AS to maintain their stability and reliability. The use of DRPs reduces the cost of building power plants to cover peak load consumption. Optimal scheduling of MG's participation in joint energy and AS markets considering DRPs is the main purpose of this paper; Fig. 2 illustrates diagram of the suggested bidding strategy of this paper.

A. REVENUE OF AS
Equation (1) presents the total revenue of AS and contains two parts. Part one describes the profit of the contract regardless of the call for AS. Part two describes the profit of calling VOLUME 10, 2023 and generating AS [25].

B. COST FUNCTION OF GENERATORS
For optimal planning of MG, the operating cost of the generators must be calculated. Equation (2) presents operating cost of MT [33]. The operating cost of these generators depends more on the cost of fuel. Equations (3) to (4) illustrate the operating cost of PV and WT, respectively [34]. RES cost depends more on the operation and maintenance cost of the generator. b 1 , b 2 and b 3 are determined by MT supplier.

C. ADDITIONAL LOSS OF PRODUCTION REACTIVE POWER
By rising the amount of reactive power generation, generator losses increase, and at the same time, active power production decreases. Nikpour et al. [25] presents different methods for calculating this power loss. Fig.3 illustrates the capability diagram of MTs. Q mnd is the reactive power production for MT's internal usage involving water circulation pump, boiler pump motors, etc. In P N without a change in active power generation, reactive power can fluctuate between Q mnd and Q N . Generating reactive power more than Q N reduces the production of active power [4], [35]. Equation (5) presents the excepted payment of a MT that participate in the reactive power market [31].
Expected payment function (EPF) : where γ 0 is the availability price, γ 1 is the cost of loss price offer for operating in under excited mode (Q min to 0), γ 2 is the cost of loss price offer for operating in region (Q mnd to Q N ) and γ 3 is the opportunity price offer for working in (Q N to Q M ). Armature current limit and field current limit are presented in [4] and [35].   4 illustrates the capability diagram of WTs. In the first region (−Q base to Q base ), the availability cost will be paid to WT to regulate operating point. In the next region (Q min to −Q base ), and third region (Q base to Q M ), by rising the amount of reactive power, active power decreases; consequently, internal active power losses rise; hence, the cost of reducing active power production will be paid to the WT [31]. Equation (6) presents the excepted payment of MTs that participates in the reactive power market [31].
Expected payment function (EPF) : The availability price, the price offer for operating in (Q min to −Q base ) and the price offer for operating in (Q base toQ M ) are illustrated by γ 0 , γ 1 , γ 2 respectively [34].
6 VOLUME 10, 2023 where I 0 is the coefficient of the loss curves denoting standby losses. I R is current dependent loss, and I V is voltage dependent loss. In PVs, solar radiation is converted into electrical power. Equation )8( shows limitation of active and reactive power in electronic converters of PV [37]. Loss in (8) can be modeled like (7) [25], [36].

D. UNCERTAINTIES
MG's should make a decision in the presence of an array of uncertainties such as wind speed and solar radiation. In current work, Beta and Weibull PDFs are suggested to cover uncertainties of solar radiation and wind speed. This PDFs are presented in [7], [25], [19], and [38]. The output power of PV is described in [19], [25], and [29] and the output power of WT is represented in [7], [19], [25], and [38].

E. DEMAND RESPONSE PROGRAMS
In this paper, load shifting and load curtailments are modeled; both programs are incentives programs. In these models, a reward is considered for customer's participation in the DR markets; hence, the amount of participation is proportional to the amount of the reward. In the load curtailments program, consumers can recover their participation in DRPs as (44) [32]. DRPs cost can be calculated by (9) and (10). PF ls,t (s)×P ls,t × λ ls,t +PF lc,t (s) × P lc,t × λ lc,t −PF lrc,t (s) ×P lrc,t × λ lrc,t ∀s, t PF drp models uncertainty of customer's participation in DRPs. N drp is the number of DRPs and s is the number of scenarios. As incentive price increases, the amount of customer's participation in the DRPs increases. The coefficient between incentive and participation can have different scenarios between zero and one. Customer's reaction to the price changes and the relation between P drp and γ drp can be modeled using the stepwise DR function. This function and its equations are presented in [39]; Fig. 5 is one sample of this stepwise DR function that is used in this paper's simulation.

F. OBJECTIVE FUNCTION
As difference between revenue and cost maximizes, profit of MG maximizes. Equation (11) to (15) describe this objective n,s,t , Q n,s,t +λ ru × C n P E n,s,t + P ru n,s,t , Q n,s,t + λ sp − λ ns × C n P E n,s,t + P sp n,s,t , Q n,s,t +λ ns × C n P E n,s,t + P sp n,s,t + P ns n,s,t , Q n,s,t +λ rd × C n P E n,s,t − P rd n,s,t , Q n,s,t C drp = PF ls,t (s) × P ls,t × λ ls,t + PF lc,t (s) × P lc,t ×λ lc,t +PF lrc,t (s) × P lrc,t × λ lrc,t ∀s, t The cost of generating units contains five parts. The first term is the cost of participation in energy and reactive power markets. If MG's participates in any other market, the probability of calling that market will be reduced by the coefficient of this term. Therefore, the maximum of this coefficient is equal to one. The second term is the cost of participation in regulation up, energy, and reactive power markets. The third, fourth, and fifth parts are like the second part for spinning reserve, non-spinning reserve, and regulation down respectively. MGs continue to generate power for spinning reserve even by calling non-spinning reserve (the spinning reserve coefficient illustrates this point). The cost of generating reactive power must be added to the cost in all terms. VOLUME 10, 2023

G. CONSTRAINTS 1) GRID AND GENERATORS CONSTRAINT
Equations (16) to (19) limit the apparent power, active power, reactive power and voltage of each generator. Equations (20) to (24) state that participation in any of the markets cannot be less than zero for each generator [40], [41]. Equations (25) to (29) state that the total amount of MG's participation in any of the markets cannot be less than zero and more than power systems requirement. Equation (30) presents generators work at least minimum power, and cannot participate by complete capacity in regulation down market. Equations (31) and (32) control the amount of each generation unit power changes per hour [42].

2) STORAGE CONSTRAINT
In present paper, ESS just take part in the active power and AS markets. Equation (33) presents relation of storage efficiency with stored power [8]. Equations (34) to (36) curb the amount of charge and discharge power in each hour and the maximum amount of stored energy. Equation (37) limits total participation in all markets. Equation (38) describes that the ESS cannot charge and discharge simultaneously [3].
s sh + s dsh ≤ 1 (38) s sh and s dsh are binary variables.

3) DR CONSTRAINT
Equation (39) limits the maximum DR capacity. Equations (40) and (41) limit the minimum and maximum duration of the DRP. Equation (42) limits the amount of DRP changes per hour. roc max drp (t) is ramp rate of DR. Equation (43) limits the number of DR programs that run in one day. Finally, for the load shifting program, (44) illustrates the capacity participated in the DRPs must be recovered. Recovery factor (RCF) is depended to customers [32], [39].

H. RISK MANAGEMEN
For optimal planning, the risk of forecasting different variables must be considered. In previous studies, various methods have been used as solutions. Risk management in this paper is suggested by the CVaR method.
For controlling the effect of uncertainties, CVaR index can be added to the objective function. CVaR is presented as (45) to (47) [28], [43]: var − profit s ≤ η s ∀s (46) η s ≥ 0 ∀s Usually, δ is considered to be between 0.9 and 0.99. The cost function changes as follows in (48)

III. CASE STUDY
In this part, the ERCOT market is simulated with real-world data on 27th October 2020. It is assumed that participation in the reactive power market is annual, and MG must generate a total of 300 kVAR of reactive power per hour. DRPs partnership is only in the EM. Finally, participation rate of each unit in each of the markets, incentive price, amount offered in DRPs, and amount of energy storage per hour are obtained. For comparison a same simulation is done for 12th July 2020; outputs of this simulation clarify the same idea of first simulation.

A. DATA OF MG
The proposed MG possess two MTs, one WT, one PV and ESS. Table 2, table 3 and table 4 present the specification of these units. The specifications of DRPs are in tables 5. Fig. 5 illustrates the relationship between incentive price and customer's participation in different DRPs. It seems that by increasing the incentive, customer's participation in DRPs will increase.

B. DATA OF MARKET
ERCOT coordinates transactions between competitive power sellers and buyers. In addition, it collects money from companies that consume power and pays the resources that produce the power. Energy and AS requirement information is published each morning at 6 o'clock in the ERCOT market. Participants have time until 10 o'clock to offer their bidding in different markets. The market operator then publishes the results by 13:30 [45].  Fig. 6 depicts price of energy and AS in ERCOT for day of the case study [45], [46]. According to this figure, the prices of energy (from 8 to 12) and spinning reserve (from 13 to 19) are higher than other ones.

2) PROBABILITY OF CALL AS
Usually, database of electricity markets contains information about hourly energy consumption, average energy consumption, number of hourly AS calls, etc., [45]. Probability of call AS is calculated as (49). Probability of call AS for ERCOT market in 2019 for regulation up is 0.0069; for regulation down is 0.0067; for spinning reserve is 0.055; for non-spinning reserve is 0.032.  Fig. 7. When the wind speed is lower than v in , the output power of WT will be zero; so, the first scenario of the WT is different from other ones. When the wind speed is between v r and v rout , the output power of WT will be maximum and constant like the scenario number six (these points are available in [25] Eq. 24). These scenarios are generated by using different PDFs (Beta and Weibull). Weibull: α = 6.6663, β = 9.6131, γ = 0 Beta: α1 = 0.61, α2 = 225, a = −25 b = 54096 VOLUME 10, 2023   Fig. 8 illustrates the MG generates 300 kVAR reactive power per hour according to the annually contract. The second MT has the largest participation in this market due to its higher operating cost than other generators in MG. AS markets are the first choice of units with high operating cost. In these markets, generators generally do not work at full capacity. Hence, their operating costs are reduced, and their profits are maximized.
According to Fig. 9, MT1 has the highest participation in the EM in the first hours; however, in the following hours, with the increase in the price of spinning reserve market, it increases its participation in this market. With a decrease in the price of spinning reserve market, MT1 increases its participation in EM. According to (30) and table 3, it is clear that this generator cannot take part in the regulation down with full production capacity.
As shown in Fig. 10, in the morning (the price of energy is not high), MT2 provides power for the DR recovery program. With the increase of the energy and spinning reserve markets prices, the participation of this generator in these markets will  increase. In some hours, such as 7 and 13, the prices of energy and AS are cheap; though these prices in the following hours increase; hence, MT2 generator performs energy storage. According to Fig. 11, PV reduces its participation in EM with the slope of energy price reduction, and stores its energy (selling this stored energy in following hours ( fig.13) has more profit for the MG). As a result of increasing the price of spinning reserve market, PV takes part in this market.  Due to the low operating costs of renewable generators, these generators are more inclined to EM. So, as presented in Fig. 12, before 10 o'clock, WT stores its energy by considering ESS constraints to sell it with increasing energy prices. In 10 to 12 o'clock (during the pick price of EM); in addition, to selling stored energy, WT offers its production capacity in the EM. By reducing the price of energy, WT stores its generated energy to sell in the spinning reserve market at 16 to 18 o'clock.
According to fig.6, at 16 o'clock, the price of spinning reserve is about 4 times higher than the price of energy at the same time. If the participation of the MG is limited to EM, the profit of the MG is about 140$. However, according to the results, the profit of the MG increases from about 140$ to about 650$ (about 4.6 times) by simultaneous participation in energy and spinning reserve markets. This fact has two reasons: cost reduction due to power reduction in AS markets (according to probability of call AS), and the payment for capacity reserve.
According to Fig. 13, when energy is cheap, energy storage is done. Then, MGs sell the stored energy at expensive hours. The energy losses and high prices of the storage devices are their drawbacks; however, the difference in energy prices at different hours ignores ESS losses. With a quick view to Fig. 14, load shifting is run from 8 to 13 o'clock, and load curtailments is run from 9 to 13 o'clock when the price of electricity is at its maximum. Exactly equal to Fig.10, DR recovery programs is run from 2 to 5 o'clock.
Both programs have been run within the maximum allowed time until the MG reaches the maximum possible profit. The incentive prices of both programs are in about the final stages, due to the high price of energy, it is profitable for MG (Table 6 shows the bidding for incentive prices; Fig. 5 illustrates the relationship between incentive price and customer's participation in different DRPs).
Typically, if the selling price of energy is more than the incentive price of DRPs, the MG will use DRPs. From 2 to 5 o'clock, when the prices of electricity are cheap, DR recovery program is run. In calculating the cost of DRPs, the DR recovery program cost should be considered.
As it was mentioned at the beginning of part III, a same simulation is done for 12th July 2020. The data (weather, price, etc.) of this day and this simulation is available in [25]; however, the generation units are the same as the generation units of the first simulation. Fig. 15 and fig.16 present a same idea as it was presented in the first simulation. The MT of this simulation (Fig. 15) just participates in EM with its minimum power (according to its table in [25]) except in 15 o'clock (the price of energy is high at this hour); in other hours, the MT prefers to take part in regulation up market like the MTs of the first simulation.   EM market has more profit than AS for the WT of the second simulation (Fig. 16) due to the low operating costs of renewable generators. So, this generator participates with all of its capacity in EM like the WT of the first simulation.

IV. CONCLUSION
This paper presents a novel model for microgrid (MG) optimal scheduling strategy in coupled active, reactive, and ancillary services (AS) markets. A combination of AS, DRPs, and ESS was used to increase the security and the stability of the power systems considering uncertainties of RES, risk management and additional loss of reactive power generation. The results demonstrate that the generators with high operational cost can be more interested in participating in AS markets. The call for deploying AS factor presented that generation units are not active in the whole moments of one hour with complete capacity in AS market. Hence, their operational cost falls; they can increase their revenue, especially for high-cost generators.
By reducing the probability of calling AS, the run time of generators declines; so, expensive units became more interested in AS market. On the other hand, cheaper generators participated more in EM. The mathematical equations and the capability curve present that if the amount of reactive power generation increases, their loss will increase; consequently, MGs divide reactive power production among different units in peak hours. In off-peak hours, MGs tend to store energy and sell it at higher prices in peak hours. Due to the efficiency of ESS, part of the energy will be wasted; however, according to the difference between energy prices in different hours, usage of ESS becomes efficient. MG used DRPs when the energy price was high, and performed a recovery program when the energy price was at the lowest point. In addition, because of the high price of energy, MG can suggest high incentive to participators.