A Risk-Based Framework for Power System Modeling to Improve Resilience to Extreme Events

The extent of the damage to Puerto Rico from Hurricane Maria in September 2017 led to outages in electricity service that persisted for months. Power system operators attempting to restore critical facilities faced challenges on almost every front, from supply chain interruptions to the inaccessibility of key assets. After a disaster of this magnitude, it is critical, but challenging, to prioritize how limited resources are directed toward rebuilding and fortifying the electric power system. To inform these decisions, the U.S. Department of Energy funded efforts investigating methodologies to identify critical vulnerabilities to the Puerto Rican power system, and to provide data-driven recommendations on how to harden and operate the system for greater resilience. This work presents the Risk-based Contingency Analysis Tool (RCAT), a framework developed as a part of that resilience initiative. The framework can qualitatively and quantitatively describe the most critical system vulnerabilities with an understanding of both likelihood of occurrence and impact. It evaluates the effectiveness of candidate remediation strategies in reducing overall risk to the system from future hurricane events. This paper will describe RCAT, with an emphasis on how different modeling capabilities have been integrated along with probabilistic methods and analytical metrics to better describe risk.


I. INTRODUCTION
I N SEPTEMBER 2017, Hurricane Maria struck the island of Puerto Rico and became the third costliest hurricane in U.S. history and by far the most destructive to hit Puerto Rico in modern history [1]. Coming in the wake of Hurricane Irma, Maria's impact on the island's electrical infrastructure was devastating and resulted in interruptions to electrical service that, for some parts of the island, continued for months. Following an event of this magnitude, restoring the system as well as planning and hardening it to protect against future events are huge undertakings. Given the time and expense associated with these measures, the resources available to improve the resilience of the power system are always a driving constraint, and so it is critical to have sound, datadriven methods for prioritizing these efforts. However, it can be challenging to evaluate the relative benefits of different infrastructure upgrades or to compare these with different operational planning strategies. This challenge is intensified VOLUME 10, 2023 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ by the fact that studying extreme events is outside of common industry practice, where planning studies tend to focus on ensuring reliability to a set of common or credible contingencies.
The increasing frequency and severity of extreme weather are drawing more attention and resources to preparing power systems to withstand these types of high impact, low probability events [2], [3], [4], [5], [6], [7], [8]. This has led to the many new approaches to assist in both power system planning and operations that would enable greater system resilience. These include the development of new metrics [9], [10]; new analytical methods for outage prediction [9], [12]; optimal placement, sizing, and design of microgrid systems [13], [14], [15]; evaluating the vulnerability of generation supply mix to different extreme weather conditions [16], [17], [18]; and the development of new methods to model the impact of natural hazards on power system infrastructure [19]. Related to hurricane threats in particular, work has been done to characterize power system fragility and simulate cascading faults caused by strong winds [20], [21].
Pacific Northwest National Laboratory (PNNL) has developed an integrated modeling framework capable of applying some of these approaches and evaluating system performance using physics-based engineering models to quantify system resilience at varying levels of aggregation. This modeling framework, called the Risk-based Contingency Analysis Tool (RCAT), was developed to evaluate bulk electric system infrastructure upgrades and operational decisions based on their impact on power system resilience. The framework of this work is an extension of the work presented in [22].
One of the primary contributions of this work is the development of new approaches and metrics for quantifying system health and resilience, both at the system level and decomposed to inform specific functions and decision making. The innovation lies in the combination of existing tools and modeling capabilities to create an integrated approach to modeling the effects of extreme weather events, coupling these approaches with new metrics and applying data-analysis techniques, including clustering, to rank asset criticality. A power system asset fragility tool is combined with a probabilistic Monte Carlo model, a dynamic contingency analysis tool, and a set of novel metrics to deliver a complete end-to-end simulation of extreme weather event to mitigation strategy to outcomes, which returns actionable information for relevant stakeholders such as utilities.
In this paper, Section II provides an overview of RCAT, including a brief discussion of the key tools that were leveraged or adapted. Section III describes the new methodologies that were developed, including a probabilistic method for driving the risk-assessment, and geospatial mapping and temporal sequencing to understand how a weather event will impact the system over time. Section IV describes the multilevel metrics and analysis approaches that were created to support the interpretation of results and to enable decision making. Section V describes how these metrics can be used to inform decision making, both within the infrastructure and operational planning contexts. Finally, Section VI provides conclusions.

II. TOOLS LEVERAGED WITHIN THE FRAMEWORK
At its core, RCAT is a framework that enables risk-based contingency analysis to support decision making at both the long-term planning and operations timeframes. Fig. 1 provides a high-level diagram of its core functions. Several different data sets, models, and systems analysis methods are required to support this decision-making capability, and the framework has been built to leverage existing best-in-class tools.

A. ELECTRICAL GRID RESILIENCE AND ASSESSMENT SYSTEM
The Electrical Grid Resilience and Assessment System (EGRASS) is a web-based geospatial application that was originally developed to recommend candidate technology deployments that improve distribution system resilience in service of critical end-use loads [23], [24]. In support of RCAT and applying EGRASS to power system transmission networks, new features were developed to provide linkages between disparate data sets, including grid topology, censusbased population, reliability indices, location of electrical grid critical infrastructure, and other relevant geospatial data. This was particularly critical in defining probabilities of failure of assets due to high winds from a hurricane as well as the temporal dimension of the contingency scenarios. While other models like [25] estimate failure probability of assets, the sequence on which assets fail and the time in between failures is not estimated. These factors are very important for the RCAT framework to enable high-fidelity dynamic cascading failure grid studies in the Dynamic Contingency Analysis Tool (DCAT). More detail on these methods is given in Section III.

B. DYNAMIC CONTINGENCY ANALYSIS TOOL
DCAT [29], [30] is a software capability that enables the modeling of major disturbances on the grid and captures the behavior of the system under events that could lead to cascading blackouts. Leveraging commercial power system planning tools (in the case of this analysis, Siemens PSS R E), DCAT uses a hybrid dynamic and steady-state modeling approach to simulate cascading-outage sequences.
In the context of RCAT, DCAT is used to evaluate system performance for each extreme event outage scenario and to quantify system health following the event. Many traditional power system health metrics are used in this analysis, including loss of service to end-use loads, deviations in voltages, and overloading on transmission lines and transformers. Under this work, some additional key metrics were developed using outputs of DCAT simulations across many extreme event scenarios to better quantify risk and system resilience. These metrics are discussed in greater detail in Section IV.

III. METHODS AND PROCEDURES
At the highest level, RCAT is a probabilistic power system analysis tool that can be used to run hundreds or thousands of possible outage scenarios to establish a statistical understanding of probable impact across a likely threat landscape. RCAT can be driven by either historical data, as discussed here, or by forecasted threat information. Once the analysis has been run, key vulnerabilities and the most probable outcomes can be identified, which then inform mitigation measures. These mitigation measures could include changes to system operation, which are likely to be more useful in the context of near-term threat forecasts, or upgrades to system infrastructure, which could feed into long-term planning efforts.
For the purpose of exploring the methodologies leveraged within RCAT, this section will consider the architecture in two layers: the full workflow of the analysis, which includes simulations of many stochastic scenarios, and an individual simulation workflow. It will begin with a brief overview of the full analysis framework, provide more detail on the workflow of a single run, and then end with a discussion of how the results are considered when developing metrics and identifying mitigation measures.

A. ANALYSIS FRAMEWORK
RCAT leverages a physics-based model of the power system and a threat model capable of translating historical or predicted threats into power system failure probabilities. These models are used to stochastically determine a suite of credible outage scenarios to drive a statistical representation of system performance in response to the modeled threat. Fig. 2 provides a simplified flow diagram detailing each major capability, including its required data inputs and produced outputs. RCAT's data requirements are intensive and extend beyond what is standard in the industry. Significant work was therefore required to not only develop or collect the requisite data, but also to develop the appropriate data linkages to enable the tools and methods deployed here to work within a single analysis. This effort, along with recommendations to inform industry standards to better facilitate this work moving forward, are detailed in [31].
While Fig. 2 depicts the workflow as linear, in reality the mitigation measures determined by running the analysis can then be incorporated into the system model, allowing the analysis to be rerun to evaluate the effectiveness of those measures to improve system performance.
A key contribution of this work is the development of statistical analysis methods and risk-based performance metrics to quantify system performance across these probabilistically created outage scenarios.

B. SYSTEM AND THREAT MODELS
A key component of RCAT is the ability to run physicsbased models to simulate system behavior under a modeled threat. A flow model of the power system under investigation is therefore a fundamental requirement of the framework. In the case of the Puerto Rico analysis, several PSS R E planning cases were provided by the Puerto Rico Electric Power Authority (PREPA). These models were developed during the 2019 integrated resource planning process [32] and depict the system under a variety of configurations, including 2019 day and night peak loading cases, as well as anticipated grid states for 2025 and 2028 under both day and night peak loading conditions. These models are bus-branch representations of the power system and include dynamic models for generators within the system, allowing for steady state alternating current (AC) power flow and transient stability studies to be run.
In addition to the data required to run a power flow study, geospatial information for critical power system assets is necessary to map between the threat model and impacts to the power system infrastructure. For the Puerto Rico analysis, these data were pulled from geographic information system (GIS) files provided by PREPA, which had been mapped to critical substations in the power flow model. Additional geospatial mapping and network analysis was performed to map the rest of the power system to the appropriate geospatial coordinates. More detail on this work is provided in [31] and [22].
As a framework, RCAT was developed to be generic to the threat model used, provided that the threat can be represented as the probability of failure for each asset under threat in the power flow model. In the case of the Puerto Rico work, the analysis was done on historical hurricane data, where EGRASS was used to calculate the failure probabilities for transmission lines, transmission towers, substations, and power plants as a function of hurricane wind speed [28].
EGRASS uses historical hurricanes data from the National Hurricane Center (NHC) [26], [27], including Hurricane Maria, Hurricane Irma, Hurricane Lenny, Hurricane San Felipe, and others. EGRASS provides the probability of failure for specific assets in the power system using estimated fragility curves of transmission assets. The fragility curve assets can be validated in collaboration with utilities as in the process of [28].

C. OUTAGE SIMULATION
Embedded in each RCAT analysis is the simulation of hundreds or even thousands of different probabilistically determined outage scenarios. This is the process that is contained within the ''Outage Simulation'' step in Fig. 2. This process has also been incorporated into EGRASS. The results of these simulations, including system performance metrics, are then aggregated to get a better understanding of probable system performance as well as worst-and best-case scenarios. At the highest level, there is the overall probabilistic representation of the event. In the Puerto Rico analysis, that would be a hurricane event. Within each event, multiple Monte Carlo scenarios are simulated. Finally, within each scenario, groups of contingencies are simulated in scenario stages, which are essentially groups of outages contained in a particular timestep. At each level of this hierarchy, metrics on system performance are calculated and can be leveraged for analysis. More detail on these metrics and the insight they provide at each level in the hierarchy are provided in later sections.
Functionally, the outage simulation capability can be broken into roughly three steps: (1) development of a threat timeline to enable outage sequencing; (2) probabilistic sampling to generate individual outage scenarios; and (3) physics-based simulation of each individual outage scenario, culminating in the calculation of performance metrics for that scenario. Additional detail on each of these steps is provided in later sections of this paper. Note that while Fig. depicts three failure scenarios being simulated in parallel, this is for the sake of simplicity; in reality, hundreds or thousands of failure scenarios should be run to provide a sufficient statistical sampling of the probabilistic scenario space.

D. TEMPORAL SEQUENCING
The outage probabilities output by EGRASS, much like from many threat models, are a function of the intensity of the threat-in the Puerto Rico case, hurricane windspeed. The order in which assets in the power system fail as well as the elapsed time between outages will impact how the power system responds, and it is of interest to simulate high-fidelity dynamic cascading failures in DCAT. A methodology was therefore developed for RCAT that leverages the underlying threat data, in this case the NHC hurricane data, and the geospatial location of each power system asset to establish a timeline for the outage probabilities.
For the Puerto Rico analysis, the temporal sequencing model uses 'best-track' GIS data take from the NHC dataset, which includes the central location of the hurricane and the edges of the 34-, 50-, and 64-knot wind fields at 6-hour intervals [26], as well as the maximum windspeed [27].
First, the wind field data is linearly interpolated temporally from the 6-hour resolution to the granularity required for the analysis. In the context of the Puerto Rico analysis, a 15-minute resolution is used. Linear interpolation is also used in the spatial dimension across the wind field edges to approximate windspeed at each asset's location. Any assets that fall outside of the 34-knot wind field are not considered to be impacted at that timestep. Fig. 4 provides a graphical representation of this process.
An asset is considered impacted and included in the probabilistic sampling for a timestep if the windspeed at the asset's location is above a defined threshold. For Puerto Rico, sensitivity analysis was run looking at a variety of different windspeed thresholds.

E. PROBABILISTIC SAMPLING TO DETERMINE OUTAGE SCENARIOS
A Monte Carlo simulation is used to generate staged asset outages for ''n'' sets of contingency scenarios based on the probabilities of asset failure and the temporal sequencing model previously described. The output of this method is ''n'' sets of outages, where the outages within each set are grouped based on the temporal resolution of the analysis. Once an asset fails, it is assumed to remain out of service for the duration of the event and therefore cannot fail again.

F. PHYSICS-BASED POWER SYSTEM SIMULATION
Once each outage scenario has been determined, that scenario script is then translated into a data format that the physicsbased power system simulation tools can ingest. In the case of the Puerto Rico analysis, the PSS R E version of DCAT was used to model power system behavior and capture possible cascading failures that result from the hurricane threat.
As previously discussed, and further detailed in [29], DCAT uses a hybrid dynamic and steady-state approach to simulating cascading-outage sequences. By switching between dynamic and steady-state power flow solutions, the tool models important dynamics and protection action, which can only be captured within a dynamic simulation, while still representing how the state of the system will evolve over a period of hours, which is often the duration of cascading failures or large-scale natural hazards.
Once the system's frequency has settled back to steady state, a static AC power flow solution is extracted from the results of the dynamic solution. The static power flow case is then evaluated for transmission line overload and voltage violations. If violations exist, a corrective actions modeling methodology is applied to the case to approximate both automatic and operator-determined measures to improve the health of the system.
Corrective actions are modeled using the PSS R E AC corrective actions function, which is part of the Multilevel AC Contingency Computation (MACCC) application. These include automatic actions such as transformer tap changes, switching of shunt reactors and capacitor banks, and adjusting the controls on phase shifters, static compensators and static var compensators. Additionally, the MACCC application also handles generation redispatch and load shedding, which represent the actions that an operator might take to alleviate line overloading.
Following the application of the corrective actions tools, if there are still overloaded lines within the system, DCAT will select the line with the highest overload percentage to be tripped. Modeling this line-tripping is handled once again in the dynamic domain. This process is then repeated, including the application of corrective actions and line tripping, until the system reaches a state where no violations exist. For the Puerto Rico study, this entire process occurs for each hourly timestep and, once complete, outages for the next timestep are applied and the process repeats itself until either the complete VOLUME 10, 2023 hurricane scenario has been run or the system experiences a catastrophic failure and the simulation is terminated.

IV. METRICS AND ANALYSIS METHODS
Metrics for analyzing system performance within RCAT can be divided into four categories: (1) per-stage performance metrics that provide insight into the system's response to a particular stage, essentially representing a temporally clustered set of outages within a large scenario; (2) per-scenario performance metrics that provide insight into the electric grid's resilience to a particular outage scenario; (3) per-event performance metrics, which are an aggregation of the perscenario performance metrics for a single modeled event (e.g., a historical hurricane), weighted by the probability that an event will occur; and (4) asset-level performance metrics that provide the vulnerability and criticality ranking of specific asset failures based on the aggregated performance across many simulated cascading event scenarios. This section will cover metrics from across all four of these categories, including a discussion of how metrics are aggregated at each level.

A. PER-STAGE AND PER-SCENARIO METRICS
The per-stage performance metrics are really a subset of the per-scenario performance metrics and are calculated and collected as discussed directly below. The advantage to having these more granular results is in understanding how the system will perform over time, particularly as these types of events, such as hurricanes, often unfold over a matter of hours. They are also used in calculating asset-level metrics. The per-scenario performance metrics are the aggregation of per-stage metrics and represent the overall performance of the system for the entire sequence of stages in the scenario. In both cases, the metrics themselves can be derived directly from the physics-based DCAT results. These metrics are given in TABLE 1.
In addition to the standard metrics described above, a more comprehensive metric was developed to quantify impact to load in the system, inclusive of both complete outage and compromised performance due to voltage violations at the load bus within the power flow case. This metric will be referred to as ''load impact'' and is calculated as described in Eq. 1-3. Fig 3. provides a graphical representation.
These per-scenario metrics provide insight into how the system would respond to a particular outage scenario. In cases where a particular sequence is of primary interest or the anticipated outages are well-understood, these per-scenario results may be sufficient. However, when considering forecasted events with a high degree of uncertainty, or when trying to understand the system's performance under different manifestations of a credible threat, many scenarios must be considered and incorporated into the performance metrics.
where P V ,bus−x is piecewise linear function of voltage deviation as described in (2).
where V bus−x is the voltage magnitude at Bus x.

B. PER-EVENT METRICS
The next level of aggregation is per-event metrics, where system performance is captured across many different probabilistic realizations of the same event. This captures the uncertainty associated with how a particular event will impact the power system and, when enough scenarios are studied, should provide a distribution of system performance. For event-level metrics, the same performance metrics are considered as described at the scenario level; however, rather than considering a single number, each metric is treated as a distribution. Statistical analysis can then be performed on these metrics to capture the average, worst-case and best-case performance of the system.

C. ASSET-LEVEL PERFORMANCE METRICS
By aggregating the results from across different simulation stages, scenarios, and events, and then statistically decomposing those impacts such that they can be attributed to failures at the component level, it is possible to begin to quantify the criticality of specific assets. These asset-level performance metrics are particularly valuable in identifying component-level vulnerabilities and informing where infrastructure reinforcements, upgrades, or additional redundancies could provide the most value in improving system resilience.
The process for calculating these asset-level metrics requires new statistical methods as described below and in [33].

1) AGGREGATED IMPACT LEVEL
The process starts by calculating the aggregated impact level (AIL) of a set of contingencies occurring in a single stage of a scenario based on the resulting performance metrics at that stage. For each hurricane h, a binary failure matrix F h with N A rows and N S columns is prepared based on occurrence of assets failures, where N A is the number of assets in the system under consideration for criticality, and N S is the number of stages in which observed performance metrics are available from hurricane h simulation. For analyzing the overall impact of a single contingency or group of contingencies occurring together, all of the scaled performance metrics are considered as features for each contingency, and are analyzed using a clustering process that groups the contingency events based on similar impact levels. K-means clustering was applied on these scaled features of all the contingency sets, with an optimal number of clusters selected from Silhouette scores [34]. Each resulting cluster represents a specific pattern of impacts based on a specific distribution of metrics in the cluster. Since each contingency stage involves a set of asset failures, the clustering process identifies which groups of asset failures have a similar level of impact on the system. The impact pattern of resulting clusters can then be ranked based on severity. This ranking indicates the relative AIL of each set of contingencies. The mean of the cluster centers formed using the scaled impact metrics can be used to determine the AIL of the cluster. Weighted means, though not used here, could be considered in cases where certain impact indices are prioritized in the computation.
A higher AIL is indicative of greater impacts to the system. Fig. 6 presents the results of the asset-level metrics, with all sets of asset failures/contingency events grouped into five clusters denoted by AILs ranging from 0-4, where 0 is the group with the lowest impact AIL and 4 is the group with the highest. Each dot in Fig. 6 represents a group of asset failures occurring within a stage, and the position of the dot represents the resulting impact of that group of failures at that stage. A set of asset failures occurring at any stage is considered severe when it falls in cluster AIL-4.

2) TRANSITIONAL IMPACT LEVEL
AIL broadly represents a pattern of impact metric values. Note that a contingency consisting of the same group of asset failures will have a different impact, and therefore a different AIL, depending on the state of the system when the asset failures occur. In other words, the stage, scenario, and overall event will cause the AIL of the same contingency set to vary. Since the AIL is indicative of the impact on the system at the end of a particular stage, it also indicates the cumulative impact from all previous assets failures. Looking at only AIL is not generally sufficient to rank asset failures based on their severity. Another new impact metric has therefore been developed, the transitional impact level (TIL), which considers the change in AIL from the previous to the current stage due to a group of asset failures in the current stage. Monitoring TILs for an asset or a group of assets affected in a stage of the event, as well as across Monte Carlo scenarios, can help determine the criticality of an asset based on its immediate impact on the system. TILs can be quantified as follows.
Consider a hurricane scenario where the k-1st stage resulted in AIL-x (pre-AIL). Then, at the k th stage, a set S k = i : i ∈ (1, N A ) |F h i,k = 1 of asset (i) failures drove the system to AIL-y. The TIL due to the asset's failure is defined as (y, x). When the total number of AILs identified from clustering is N AIL , the number of possible TILs for the system becomes N TIL = 0.5(N 2 AIL −N AIL +2). Each hurricane scenario evolves into different TILs during its propagation.
A TIL matrix T with N A rows and N TIL columns is prepared, whose elements t i,j indicates, for each asset i, the number of hurricane scenarios in which the asset i failure has caused a specific pair of TIL values (y j , x j ) at any stage across all hurricane scenarios. This information is then used to correlate the causality of specific impact transitions with individual asset failure occurrences. The difference between these AILs (y j − x j ) indicates the impact severity of the asset failures. If, out of all occurrences in different hurricane events, the failure of an asset causes severe impact transitions with a high degree of frequency, then that asset can be considered to have a severe impact.
Consider a simple example where only loss of load (LOL) is considered. During a hurricane progression, first a few lines (A 1 , A 2 , A 3 ) are tripped, which causes an LOL of 1 MW and is labeled as a low-impact pattern AIL-0. As the hurricane progresses, buses or lines (A 5 , A 6 , A 7 ) are tripped, which causes 500 MW of additional LOL and is labeled AIL-4. This implies the asset failure (A 5 , A 6 , A 7 ) has caused the system to jump from AIL-0 to AIL-4, which is considered a severe impact transition pattern of (y = 4, x = 0).
Given the complexity and non-linear nature of the power system, the usefulness of these metrics when considering a single set of results is limited. However, across many simulations and by including a diverse set of fundamental metrics (i.e., in addition to simply LOL), these metrics begin to illuminate patterns that can lead to a better understanding of asset criticality. For example, assume that asset failures A 5 , A 6 , A 7 occurred in 100 hurricane simulations, causing a high-impact transition (4,0) 95 times. That means A 5 , A 6 , A 7 has a high (sample) probability (95/100) of causing a high TIL.
From T, probability of occurrence of all TILs can be calculated in matrix F(N A × N TIL ), whose elements as defined in Eq. 4 represent sample probability of occurrence of TIL (y j , x j ) upon the event of asset failure A i .
It is assumed that all individual assets in a group of simultaneous failures have a similar contribution to the performance metrics seen at the end of that stage. Since any of those assets failing together might have more or less impact than others in the group, the uncertainty in the causality of impact from a particular element, U i , must be accounted for. A uniform probability, P(U i ) = 1/N a can be considered while calculating f i,j , where N a is the number of assets in the group that are failing together in a scenario. Given enough Monte Carlo scenarios, it is possible to assess the impact of an individual asset failure by using enough hurricane simulation stages along with transitional AILs.
Any asset failure that has a high probability of causing high-impact transitions (TIL) and a higher probability of occurring frequently across hurricane scenarios is considered a critical asset. The following subsections (iii-iv) present the proposed ranking metrics to quantify these probability comparisons.

3) VULNERABILITY RANK
Vulnerability rank R v (A i ) of an asset A i is defined as how vulnerable the asset is to fail during the hurricane H . If n(H ) represents the total number of hurricane scenarios, and n(A i ) represents the number of occurrences of the asset failure A i in the hurricane simulations, then the sample probability of occurrence of event A i is defined as P(A i ) = n(A i )/n(H ). Higher P(A i ) indicates the asset is more vulnerable and a vulnerability ranking metric (VRM) v(N A × 1) with v i = P(A i ) can be prepared for vulnerability rank calculation.

4) CRITICALITY RANK
Using the transition in impact levels, the criticality of an asset failure can then be calculated. Since a given asset failure can result in different impact transition levels depending on when it occurs and on the operating conditions, a criticality ranking metric is calculated for each asset. The criticality rank of an asset is defined as the probability of it causing high-impact TILs upon failure. The high-impact TIL pairs (y j x j ) are first identified in a set as given in Eq. 5. Then the criticality ranking metric (CRM), c(N A × 1) of each asset, is calculated using Eq. 6 by summing the probability of occurrence of transitional impacts (y-x) due to the asset failure.
After determining VRM and CRM (v i , c i )∀i ∈ [1, N A ] for each asset and scaling those between (0,100%) for all assets, the following ranking algorithm is proposed to determine the vulnerability and criticality rank of an asset R v (A i ), R c (A i ).

R(
Given the ranking metric of an asset r i ∈ (v i , c i ), ranks R v (A i ), R c (A i ) are estimated as given in Eq. 7. The algorithm for computing the ranking metrics and identifying critical assets are presented in the flowchart in Fig. 5. Fig. 8 represents the results obtained from applying the proposed algorithm for identifying critical assets during hurricane simulations. A total of N a = 59 branch and N a = 92 bus failures were involved in the Monte Carlo simulations of the hurricane scenario. The results indicate there are five patterns of criticality that exist among buses and six among branches. Additionally, one bus and one branch are highly critical across all these simulations because they have a higher probability of failure and cause most impact upon failure.

D. OVERALL SYSTEM PERFORMANCE METRICS
The final level of aggregation is the overall system performance to a particular threat. This involves studying the system's behavior across many scenarios for many events, and then aggregating and weighting the per-event performance metrics based on the probability of occurrence. This method therefore captures both the impact of each event and the likelihood of the event, to understand the biggest risks to the system.

V. UTILITY OF RESULTS IN POWER SYSTEM DECISION-MAKING
The metrics outlined above, including the analysis approaches developed both to obtain the requisite data and to quantify them, are one of the primary contributions of this work. While some of these metrics are building blocks for calculating other, higher-level metrics, most can be used to assist decision making under certain planning or operational scenarios. With the emerging roles of distributed energy resources in the grid reliability and resilience evaluation, as well as more strict requirements from end users, the proposed load impact metric can help account for the impact of voltage volatility. This composite metric can assist grid planners and operators to gauge the severity of system damage and electricity service concisely.
While the per-stage and per-simulation metrics are primarily used to build other, more comprehensive metrics, they also provide more granularity into how events will unfold and how outages will be sequenced. Future work might investigate how cross-simulation analysis at each stage could yield insight into probable patterns of damage and loss of service within the system, which could help in staging emergency response activities and inform operator dispatch decisions when a major event is forecast.
Per-event metrics provide insight into how to plan for a forecasted event. Running many Monte Carlo simulations to understand the best, worst, and most likely impacts of an anticipated event bounds the most likely outcomes, and can inform how to the system should be operated to mitigate these impacts and how to stage response activities.
Per-asset metrics can be used to quantify the vulnerability and criticality of individual assets within the power system. As in the case of the per-event metrics, this could help in nearer term decision making by informing operating strategies to proactively reconfigure and redispatch the system and help prioritize restoration and response efforts. Over the longer term, this enables risk-based decision making in evaluating investments that could reinforce or introduce redundancy for the most critical assets.
Future work could focus on developing methods for establishing correlations across asset failures. This would help operators understand where new points of criticality exist as parts of the system are taken out of service. For example, the loss of one component could make the failure of another either more probable or more critical. Studying these correlations could further inform operational decision making by giving operators better and faster insight into where to focus mitigation efforts as portions of the system fail during major events.
Aggregated overall system metrics for a particular threat type help in evaluating large-scale upgrades and understanding overall system resilience. One application of this framework is in performing cost-benefit analyses for proposed infrastructure upgrades. This can simulate system performance under a wide range of threat scenarios, incorporating different possible reinforcements and a mechanism for comparing and prioritizing how limited resources should be spent.

VI. CONCLUSION
The primary contribution of this work has been in establishing a robust framework for doing risk-based power systems analysis based on engineering models, and leveraging new analytical methods to provide insight into system performance and health. To provide a comprehensive understanding of system vulnerabilities and possible impacts, many simulations are often required to represent the range of credible threats as well as the different configurations and operational states of the power system. Even when the complexities associated with running these studies and obtaining these results are overcome, it can be difficult to distill analysis outputs into actionable information. This effort has therefore included the development of new metrics and analytical approaches to help in decision making.