Latency-Optimal Network Intelligence Services in SDN/NFV-Based Energy Internet Cyberinfrastructure

Energy internet (EI) is a very complex system with various applications that not only require a high-level of cyber-security but also need low-latency communication. Thus, cyberinfrastructure with latency-optimal network intelligence services (NIS), in which application data flows are deeply examined in real-time, is inevitable. In the future internet system, a set of NIS can flexibly be implemented in network function virtualization (NFV)-based middleboxes that overlay on software-defined networking (SDN) architecture, becoming an SDN/NFV-based cyberinfrastructure. However, how to deploy these middleboxes is a non-deterministic optimization problem, which is complicated and time-consuming. Hence, by focusing on latency minimization, we develop an artificial intelligence (AI)-powered solution consisted of two phases. First, middleboxes placement based on the graph cluster analysis, and second, NIS resource allocation based on the prediction of service usage-ratio in each corresponding cluster. The simulation-based experimental evaluation shows that our proposed strategy using an optimized K-means algorithm outperforms the recent state-of-the-art middleboxes placement approaches. The average end-to-end flow latencies are around 23.81%, 18.44%, and 11.49% lower compared with the simulated annealing method, the basic sequential algorithmic scheme, and the minimum spanning tree procedure, respectively. Besides, the proposed resource allocation scheme optimizes further the latency minimization around 4.24%. We believe that the work presented in this paper will aid the communication service providers (CSP) in providing a secure and low-latency SDN/NFV-based cyberinfrastructure for the EI ecosystem.


I. INTRODUCTION
Recently, the penetration of renewable energy generation, such as building-integrated photovoltaics (BIPV), has been increased in many countries [1]- [3]. With renewable energy generation, consumers can evolve into prosumers, a new type of energy stakeholders that can produce and use their own electricity, and also sell their excessed energy to the The associate editor coordinating the review of this manuscript and approving it for publication was Mubashir Husain Rehmani . market. Therefore, various smart grid technologies and applications have been proposed to accommodate the high penetration of prosumers with distributed renewable energy resources (DRERs) and distributed energy storage devices (DESDs) [4], [5]. These smart grid technological advancements bring opportunities to transform the current power system to energy internet (EI), an internet business model of the electricity grid, in which multiple energy and data flows are in dual circulation and coupling among the entire value chain. VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see http://creativecommons.org/licenses/by/4.0/ FIGURE 1. Illustration of EI ecosystem [5]. E-LAN is a localized group of energy stakeholders that can operate independently, or connected to the grid to buy/sell energy. Similar to the internet router, the energy router is used as an intermediate device to exchange both energy and data bidirectionally between E-LANs in the network.
In the EI ecosystem, all energy stakeholders can be joined flexibly and seamlessly to the closest energy local area network (E-LAN), as depicted in Fig. 1. Then, using the so-called energy router [6], the energy can be exchanged and transferred between one to another. Hence, the energy sharing economy [7] can be realized, which enables energy consumers to obtain the supplies directly from the nearest producers. Moreover, the cascading failure or blackout could also be resisted, which improves the stability of the whole electricity grid. It is owing to the immense development of the future renewable electric energy delivery and management (FREEDM) system [4], which considerably improves energy utilization containing all novel phases of energy generation, transmission, storage, and distribution. FREEDM system consists of some pivotal technologies such as the intelligent energy management (IEM) software, the distributed grid intelligence (DGI) software, the solid-state transformer (SST), the real-time remote monitoring, and the smart fault isolation device (FID).
It should be noted that the EI has attracted increasing attention of government and institutions in many countries. For rural electrification in Indonesia, a country with more than 17,000 islands, EI is the most promising solution to enable the internet of microgrids [8]. Furthermore, as a response to the Fukushima nuclear crisis, a large group of firms in Japan is starting to explore EI with an expectation to transform the country's electricity system with distributed energy and micro-grid integration [9]. Moreover, the EI was also proposed in Germany following a political decision to shut down all German nuclear reactors by 2022 [10]. Besides, the Chinese government and state grid corporation of China (SGCC) has proposed a proposal so-called ''global EI'' and then launches an action plan every five years [11]. Also, the EI platform has been launched in Europe as a novel strategy to achieve decarbonizing commitment by 2050 [12]. However, being in its infancy stage, EI business values and social benefits are becoming increasingly apparent with the advances in smart grid technologies. Thus, more research and development need to be performed to support the diverse and rigorous requirements of reliability, flexibility, latency, and security in the EI. To this end, the emerging technical initiative (ETI) on smart grid communications (SGC) has issued a positioning paper in 2018, which included EI as one of the eight research agenda structures [13].
Among novel future internet technologies, software-defined networking (SDN) is expected to be adopted in the building of cyberinfrastructure for end-to-end interactions across the entire value chain in the EI. Utilizing the SDN approach, both energy and data flows can be managed flexibly following the four principles, which are 1) logically centralized management, 2) separation of control, data, and energy planes, 3) programmability, and 4) open interfaces [14]. Therefore, some research works have been conducted recently to develop a framework and evaluate the performance of SDN-based EI cyberinfrastructure [14]- [16]. However, EI is a very complex system with various applications that have specific and strict functional requirements [17], [18]. Many applications, including distribution automation, load control signaling, and outage alarming, are described to require low-latency communication. Although some other applications, e.g., smart-meter data collection, are more latency tolerant, however, they need to have a high level confidentially, availability, and integrity. Thus, various network intelligence services (NIS) such as network security applications, traffic analysis elements, and deep-packet inspection (DPI) tools are indispensable to be utilized, fulfilling the required cyber-security in EI [18], [19].
In recent years, the ETI for network intelligence has worked together to support and endorse research towards embedding AI, SDN, and network function virtualization (NFV). In the context of cyber-security, taking service policies as inputs, a set of AI-powered NIS can be applied virtually in NFV-based middleboxes that overlay on SDN architecture, and they are becoming SDN/NFV-based cyberinfrastructure [20]. However, the use of this approach to secure and protect application data flows may increase endto-end flow latency significantly. Considering that the reliability requirement is defined in [21], the successful delivery of the application data flows, but with the latency higher than the defined requirement, can be considered as a failure.
Taking into account the balance between required cyber-security and low-latency communication is essential for many applications in the EI. In this paper, we present our work to provide latency-optimal NIS in SDN/NFVbased EI cyberinfrastructure. In fact, the end-to-end flow latency always depends on the middleboxes' placement in the network. However, some previous research works proved that this problem is a non-deterministic polynomial-time (NP)-hard, which is a complicated and time-consuming decision problem [22]- [25]. Hence, a trade-off optimization method is needed to achieve a heuristic solution. Among existing approaches, graph cluster analysis is the most popular AI-powered solution to solve the problem. However, the recent state-of-the-art graph cluster analysis methods [26]- [28], having at least two drawbacks that can not guarantee the end-to-end flow latency, can be minimized, i.e., 1) randomly choosing the clusters' threshold and 2) arbitrarily selecting the initial center. Hence, the main contributions of this paper are as follows.
1) We introduce the utilization of NIS for fulfilling the main cyber-security requirements in the EI ecosystem. All NIS are virtually implemented in a number of NFV-based middleboxes. 2) We reformulate an objective function for the latency minimization problem. It should be noted that for the time-critical energy control signaling application, the end-to-end latency should be less than 3 ms [6]. 3) We consider three main constraints, i.e., the middleboxes' processing power capacity, the forwarding nodes' memory resource, and the communication network configuration. These constraints are the minimum obstacles in such SDN/NFV-based cyberinfrastructure. 4) We develop the AI-powered solution, which consists of two phases. First, an optimized K-means algorithm is utilized to find the latency-optimal middleboxes placement in several clusters. Second, a prediction of NIS usage-ratio is employed to develop a dynamic resource allocation scheme, which optimizes further latency minimization in the corresponding clusters. 5) We evaluate our proposed method along with the recent state-of-the-art approaches, i.e., the simulated annealing [22], the basic sequential algorithmic scheme (BSAS) [26], the minimum spanning tree (MST) [27], and the modified BSAS [28]. The simulation-based experimental comparison is carried out on two network topologies, i.e., FatTree [29] and Abilene [30]. We expect that these two topologies are representing both layered and irregular network structures of SDN/NFV-based EI cyberinfrastructure, respectively.
We believe that the result of this work can be used as a guideline for communication service providers (CSP) to provide a secure but also low-latency cyberinfrastructure for the EI. The rest of this paper is organized as follows. In the next section, we provide related works to utilize NIS in the EI ecosystem. Section III describes the system model, problem formulation, and recent state-of-the-art methods. Section IV explains the proposed solution, section V presents our evaluation, and finally, section VI concludes this paper.

A. SERVICE ABSTRACTION MODEL
In recent years, the national institute of standards and technology (NIST) and the open smart grid (OpenSG) network task force have comprehensively analyzed all possible functional requirements of various applications for the future EI. Currently, not less than 1400 application data flows have been specified in detail, including their payload size and type, security, latency, reliability, data transmission frequency, and so forth [31]. On the other hand, several groups work together to specify the quality of services (QoS) requirements for the specific application. For example, the North American synchrophasor initiative network (NASPInet), a working group with the mission to improve power system reliability and visibility through wide-area measurement and control.
The NASPInet has contextualized five classes of data services for synchrophasor applications with specific traffic attributes. As depicted in Table 1 [32], class A is to support the needs of high-performance feedback control applications. Thus, the reliable cyberinfrastructure for this class is critically essential. It should have a fast data rate and very low latency, as well as can guarantee a high level of data availability. Furthermore, classes B and C are for the applications with less strict latency requirements such as the feed-forward estimator enhancement application and the view only appli- cation, respectively. Then, class D is to support the need for post-mortem event analysis, and class E is intended for testing, research, and development.
To understand all the requirements above and provide NIS appropriately, the use of a service abstraction model (SAM) is indispensable. Therefore, it is worth to mention a SAM proposed by G.D. Nugraha et al., in [33]. In this case, the service requirements can be represented by three sets of parameters, i.e., content, context, and resources. To be detailed, the content provides the service-related parameters such as payload size and type, maximum delay, minimum bandwidth, and so forth. Furthermore, the context serves the users/applications related parameters concerning interest, such as data transmission frequency, schedule, and location. Last, the resources supply the requirements of network service resources such as networking medium, computing power, memory space, etc. Taking advantage of this model, Fig. 2 depicts the service abstraction template for application data flows in SDN/NFVbased EI cyberinfrastructure.

B. NIS APPLICATIONS
The evolution and growth of internet technologies offer possibilities for CSP to provide better QoS, as well as develop new types of services. Hence, NIS are utilized to capture the detailed information from applications, or users' data flows, to provide the analysis of their demand and to manage the usage once deployed. Some essential applications of NIS, ranging from understanding user behavior analysis to provide intrusion detection, are listed in Table 2. Recently, many research projects have been conducted to utilize AI techniques for NIS, in terms of traffic classification [34], traffic prediction [35], accelerates service provisioning [36], intrusion detection [37], and so forth. Moreover, for securing cyberinfrastructure against intruders and other threats, some experiential networked intelligence (ENI) research projects have been started recently combining AI, SDN, and NFV. For example, the SHIELD research project, as described in [38]. They demonstrate an AI-powered framework to detect attacks using a policy-driven control loop intelligently. Adopting this framework to SDN/NFV-based EI cyberinfrastructure, we can develop AI-powered attack detection and mitigation recipes. Through intent-driven and autonomous-driving network, fulfilling the main cyber-security requirements in EI ecosystem are as follows [39] 1) Attack detection and resilience operation. It is required to monitor network traffic in real-time, detect abnormal incidents due to various attacks, and continue operations in the presence of attacks using self-healing ability.

2) Identification, authentication, and access control.
It is essential to ensure that the resources are accessed only by the appropriate entities that are correctly identified. NFV middleboxes are utilized to provide sets of NIS applications, fulfilling the cyber-security requirements in the EI ecosystem. Then, the SDN controllers (cluster and global) managed all flows using AI-driven policy automatically.
As depicted in Fig. 3, SDN/NFV-based EI cyberinfrastructure consists of SDN controllers, SDN switches, and NFV middleboxes that are utilized to securely control and forward application data flows between users/applications in EI ecosystem. However, it should be noted that a latency-optimal NIS is an essential factor for the reliability of SDN/NFVbased EI cyberinfrastructure, as mentioned in the previous section. Hence, secure and low-latency communication are both required for reliable information flowing delivery. However, these objectives usually contradict each other. Therefore, a trade-off solution of the middleboxes deployment strategy to avoid the end-to-end flow latency over than requirement threshold is indispensable.
The next section will be detailed describe the system model and the problem formulation for latency-optimal NIS in SDN/NFV-based EI cyberinfrastructure. Moreover, the detailed comparison of recent state-of-the-art solutions for latency minimization is also be provided.

III. SYSTEM MODEL AND PROBLEM FORMULATION
are the subscripts of the node couple and N v is total number of the node. Let denotes the maximum number of rules can be stored in an SDN switch S = {s 1 , s 2 , s l , . . . , s N s } ∈ V is P s , thus the number of rules that are currently stored in a switch flow tables is denoted as p s ∈ P s . If the set of VOLUME 8, 2020 all NIS is denoted as . . , c b N Bc }, then an NFV-based middlebox which supplies those services is B c ∈ V. There may be N q a number middleboxes available in the network, thus let us denote N B c as the number of NIS middleboxes where B c ∈ Q. Each middlebox has a maximum processing power capacity O b to perform a set of NIS. This processing power capacity depends on the available central processing unit (CPU) in each middlebox, which is represented in Mbps unit. Table 3 depicts example of resource allocation for NIS. Following the SAM as described in the previous section, an application data flow f can be described as f n = {source n , dest n , c b n , o n , t n }. source n , and dest n are the source and the destination nodes, respectively. c b n is the must be visited NIS of a flow's network traffic from source to destination. Furthermore, o n is the amount of middlebox processing power capacity occupies by a NIS, and t n is the daily clock periods of requested NIS by a flow. With the knowledge of all information in advance, we can generate F B c , a set of flows that requires NIS from a middlebox.

B. PROBLEM FORMULATION
Let us define end-to-end flow latency, as where, d if ,bf is the aggregate latency from the source ingress-switch to the corresponding NIS middlebox and d bf ,ef is the aggregate latency from the corresponding NIS middlebox to the destination egress-switch. The aggregate latency depends on the service processing delay α = d c bn ,o n for each NIS n, and the packet delivery time β = d v i ,v j in each link between two nodes. In a bit more details, the services processing delay and the packet delivery time estimation are described in [40], expressed as where M is the number of application data flows which request NIS. For the delay-sensitive application, the satisfaction rate follows the sigmoid utility function, as depicted in Fig. 4. Thus, a precise resource allocation strategy is an avoidable task to increase QoS. Furthermore, Z max is the maximum packet size in bit, B r is the transmission bit rate in bit/s, X v i ,v j is the distance or the length of transmission medium in meter, and L s is the propagation speed in the medium in m/s. To the best of our knowledge, the propagation speed depends on the physical medium of the link, e.g., 2 × 10 8 m/s for copper wires and 3 × 10 8 m/s for wireless communication.
It is worth to be mentioned that some existing works have proposed flow routing schemes to manage data flows in the SDN/NFV-based cyberinfrastructure. Hence, a constrained shorted path has been formulated in [41] as that is, finding a forwarding route r from a set of all routes R st that minimizes the objective function f C (r) such that the delay D(r) to be less than or equal to the threshold value D max . Furthermore, the constraints could be varied, ranging from traffic-chaining ratio, bandwidth consumption, deployment cost, energy consumption, and so on [42]. However, no matter what flow routing scheme is used, the middleboxes deployment provides the most significant effect on network latency. Hence, we need to develop a proper deployment strategy, which minimizes the total latency of each flow f ∈ F B c .

C. EXISTING SOLUTIONS
Some existing approaches have been proposed to deploy NIS middleboxes in SDN/NFV-based cyberinfrastructure with minimum latency. Moreover, Liu et al. [22] formulates the latency minimization function as s.t. x n,l = 1, ∀q n ∈ Q, ∀s l ∈ S, ∀q n ∈Q R(q n )x n,l ≤ C(s l ), ∀s l ∈ S, x n,l = e, ∀q n (e) ∈ Q, ∀s l (e) ∈ S, where x n,l , n = {1, 2, 3, . . . , N q }, l = {1, 2, 3, . . . , N s } is the binary variables to represent middlebox placement scheme in a switch s l within the set of switch S, with N q and N s being the total number of middleboxes and switches, also n and l being their subscripts, respectively. Furthermore, R(q n ) is the required resource to deploy NIS middlebox q n inside the set of Q and C(s l ) is resource capacity of each switch inside S.
To provide latency minimization, three constraints are considered, i.e., constraints (6) - (8). The constraint in (6) is to guarantee that each middlebox should be successfully deployed at a location, where x n,l = 1 denotes that middlebox q n is connected to switch s l , otherwise x n,l = 0. Furthermore, the constraint in (7) is to guarantee that the total resource demand for NIS deployment at one location should not exceed the switch resource capacity. Next, the constraint in (8) is to accommodate for middleboxes that can only be deployed in certain places. It is considered that a middlebox may require a power supply and acceleration by some dedicated platforms, which are available only at some locations.
On the other hand, to explain the type m middlebox, Vu and Kim [26] formulates the objective function for latency minimization as f is a flow from a set of data flows F m that requires NIS type m from source ingress-switch to destination egress-switch via corresponding middleboxes. Furthermore, N q,m is the number of NIS middleboxes type m in the network, o is the requested processing power, and O is the maximum processing power capacity. In this context, we have two constraints, i.e., constraints (10) and (11). Constraint (10) is the switch memory resource, which is utilized to confirm that a switch has available memory for storing new route table entries. Constraint (11) is the middlebox processing power capacity, to ensure that a corresponding middlebox has enough processing power capacity to process the NIS requested by application data flows.
Reference [26] solves the latency minimization problem with two intuitive properties. The first property is derived from [22], that it is better to deploy the middlebox as close as possible to the most-usage switches. Next, the second property is its own intuitive belief. It may better to divide network such that data flow with a set of ingress-switches are close to each other to share the same middlebox in a cluster. Therefore, they used the BSAS-based clustering algorithm as their proposed solution.
In order to determine a threshold for each cluster, the packet delivery time data between each pair of ingress-switches is utilized. However, this approach has two main drawbacks, i.e., 1) one time and randomly choosing the clusters' threshold, and 2) the arbitrarily initial center selection, not being able to guarantee the end-to-end latency to be shortened. Similarly, the MST-based clustering algorithm, as described in [27], is also very dependent on the proximity threshold, which utilized to remove network edges from the MST cluster, whose lengths are greater than the threshold value. Hence, several successive values are required to be generated [28]. However, this solution needs to run a clustering algorithm many times, which requires high resource and time-consuming to find the best-considered threshold. Hence, graph cluster analysis using a threshold method should be avoided, and a more proper approach is required.
To be more details, Table 4 presents a summary of our investigation on the existing middlebox deployment strategies to support latency-optimal NIS, ranging from probabilistic search-based to graph cluster analysis-based methods. Taking advantage of this comparative analysis, we reformulate the objective function, constraints, and considered topologies for the context of EI.

IV. PROPOSED STRATEGY
Combining both objective functions and constraints in [22], [26], we reformulate the latency minimization problem as follows, as In this problem, we have three constraints, and those are the constraint (13) - (15). Constraint (13), x N ,i = 1, otherwise = 0, is to guarantee that each middlebox should be successfully connected to any SDN switch in the network. Furthermore, constraint (14) is the middlebox processing power to ensure that a corresponding middlebox having the capacity to process the NIS requested by application data flows. Next, the constraint (15) is the SDN switch memory capacity, which utilized to confirm that a switch has the available resources for storing new rule table entries.
To solve the latency minimization problem described above, we develop the AI-powered strategy, as depicted in Fig. 5. This solution consisted of two phases, i.e., the graph cluster analysis for middleboxes placement and the dynamic resource allocation based on the prediction of NIS usage-ratio in each corresponding cluster.

A. GRAPH CLUSTERING-BASED PLACEMENT
Let a cluster K consists of the SDN ingress-switches of corresponding data flows, that is K = (s 1 , s 2 , . . . , s f ), where s f is the ingress-switch of a flow f . If S B c ∈ S is the set of ingress-switches of corresponding data flows in F B c , then to determine the packet delivery time between each pair of ingress-switches, we can calculate the shortest path (SP) delay time between them, as (16) VOLUME 8, 2020 Note that the graph cluster analysis using a threshold method should be avoided due to several reasons explained in the previous section, then a more proper solution is unavoidable. In this context, we employ another popular clustering technique, K-means clustering algorithm [44], with some modification for considering several additional conditions as follows 1) Since the objective is to find NIS middleboxes placement with minimum latency, the cluster center initialization method plays a critical role. Therefore, the initialization with careful seeding selection procedure is indispensable. 2) Moreover, the recalculated cluster center should be selected from the SDN ingress-switch. Then, the cluster refinement procedure is performed to re-assign all SDN switches to the appropriate cluster. 3) Also, the distance calculation method needs to accommodate the nodes with an indirect connection or may not physically be connected to the cluster center. Denote C = {C 1 , C 2 , C k , . . . , C K } as the set of clusters and let M = {µ 1 , µ 2 , µ k , . . . , µ K } is the nearest mean of each cluster, the default K-means algorithm usually uses to partition n p observations into K (≤ n p ) clusters in which each observations belong to the cluster C k with the nearest mean µ k , expressed as the objective function J (C, M) [44], as Since the results of partitioning in K-means-based clustering is following the Voronoi cells, the Euclidean or Manhattan distance is employed to measure silhouette value for validating similarity and dissimilarity of each point to its own cluster and other clusters [45], as where, a(n d ) is the average distance from n d -th point to other points within the same cluster, b(n d ) is the minimum of all average distance from the n d -th point to the points in each kth cluster. Let the sv(n d ) range is from −1 to 1. If the sv(n d ) is close to 1, it indicates that the corresponding i-th point lies well with the cluster it belongs. Flowchart of proposed strategy for latency-optimal NIS in SDN/NFV-based EI cyberinfrastructure. The graph cluster analysis is performed at the first step to find optimal NIS middleboxes placement in a number of clusters. Then, the second step employs NIS usage-ratio prediction in the corresponding clusters to find optimal resource allocation for each service.
As depicted in Algorithm 1, at first, we collect the SP computation between each pair of flow's ingress-switches. Then, we initialize clusters using a careful seeding initialization procedure, as described in [46]. Furthermore, almost similar to the K-medoids clustering method [47], the proposed graph clustering-based middlebox placement uses the selected node, in which the SDN switch is used as the cluster center instead of using the nearest mean, expressed as the objective function J (K , Me), as where Me = {K c (0), K c (1), . . . , K c (i)}. The center of each cluster is then updated and validated to minimize the sum of SP delay time to reach all switches in the optimal number of clusters. However, to satisfy the constraint in (15), we need to check and calculate the number of stored policies in each SDN switch. Repeat the steps until it is partitioned into optimal K sub-networks. Then, finally, put NIS middleboxes in each cluster center.

B. DYNAMIC RESOURCE ALLOCATION
After all NIS middleboxes are placed in the optimal position, we then allocate resources for each NIS dynamically. In this context, the resource allocation for each service at a particular time depends on the ratio of those services repeatedly requested by applications/users in a corresponding cluster. Taking advantage of the historical data of application data flows as inputs, the usage-ratio for the next time windows is predicted using the regression trees algorithm as described in [48].
In fact, NIS with the predicted usage-ratio higher than a particular threshold θ u is subject to be considered as one of important service, similar to VOLUME 8, 2020 Algorithm 1 Graph Cluster Analysis Using Optimized K-Means Algorithm for NIS Middleboxes Placement. Input: G = (S, E), F B c , S B c ∈ S Output: NIS middleboxes placement in K clusters 1: Step 1: Compute d SP(s i ,s j ) between each pair of switches in G. 2: Step 2: Select the number of clusters K and select nodes from S B c ∈ S as the initial center of each cluster K c using careful seeding initialization procedure as in [46]. 3: Step 3: Compute packet delivery time from each node s f to existing cluster centers as d s f ,K c (i) . Then, assign each node to the closest cluster. 4: Step 4: Update cluster centers K c to find the closest switch to each obtained cluster, where the sum of shortest path delay time to reach all ingress-switches in a cluster is minimized as expressed in (19). 5: Step 5: Calculate the number of stored policies p s for each node s f ∈ K If ∃s f ∈ K such that p s ≥ P s then K c is unqualified, exclude it then back to Step 2 6: Step 6: Validate similarity and dissimilarity of each node to its own cluster and other clusters using silhouette value measurement as depicted in (18) to define the optimal number of clusters. 7: Step 7: Repeat Steps 3-6 until it is partitioned into optimal K sub-networks. where i, j, n, and t are re-used in the remain equations to indicate the targeted cluster, index of NIS application, and the number of time-windows t that services have been operated, respectively. P ijn is the amount of middlebox processing power occupied by each NIS application in previous time-windows from N t total number of observed timewindows.
Then, the predicted usage-ratio is normalized to the range of UR ij [−1:1]. Using this information, we define the resource allocation capacity of each service for the next time-window is as where Resv ij is the percentage of guaranteed CPU allocation for a service at the previous time window. Hence, unlike the existing solution in [43], which the number of NIS is adjusted based on the incremental approach. In our approach, the NIS with a higher predicted usage-ratio obtain higher resource FIGURE 6. An example of dynamic resource allocation based on the prediction of usage-ratio, in which if the NIS application in the corresponding middlebox is considered as an important service, it will get a higher allocated CPU resource at a particular time-window. allocation in the next period and vice versa accordingly. Fig. 6 shows an example of the CPU resource allocation for a set of NIS at a particular time-window. Furthermore, to protect NIS from failures due to excess and un-predicted requests, we employ NFV-Throttle procedure [49]. When the volume of the demand exceeds the resource allocation capacity, we evaluate the fraction of the request to drop as if incoming_request ≥ max_capacity; otherwise, drop_rate = 0.

A. SIMULATION TESTBED
We implement a testbed based on the NFV infrastructure emulation platform (NIEP) [50] in two machines, and each device has 3.40 GHz eight-core CPUs and 8192 MB RAM. In more detail, NIEP utilizes the Mininet [51] and the Clickon-OSv [52] to provide a complete simulation of SDN/NFVbased cyberinfrastructure. Furthermore, we decide to use two network topologies, i.e., Abilene and FatTree, which are explained in the previous section to represent two possible architectures of SDN/NFV-based EI cyberinfrastructure. The characteristics of these network topologies are summarized in Table 5.
For the graph cluster analysis, we set our testbed with several assumptions as follows. First, the communication medium between two nodes is a copper-wires with randomly assigned losses following the normal distribution. Second, the transmission bit rate in each switch-port is 100 Mbps, but the distances between the two switches are randomly different. Third, there are five NFV-based middleboxes in each simulation, and those middleboxes could provide five kinds of NIS. Next, there are also five sets of application data flows that randomly request specific services. Last, the maximum number of clusters follows the total number of middleboxes.
Moreover, to evaluate the effects of the resource allocation strategy, we develop the following scenario. The set of application data flows, starting from 30 flows and increase one by one until 70 flows are generated 100 times, respectively. Furthermore, we record the type and the number of requested resources from each generated set. Then, 70% of recorded data are used as training data to develop the usage-ratio prediction model. Using the rest of the recorded data, the usageratio is predicted, which can be utilized further to reallocate the CPU resource for each service dynamically. The summary of the parameters setting is described in Table 6.

B. LATENCY MINIMIZATION ANALYSIS
For the first evaluation, we compare the result of our cluster center initialization strategy with the original method of the K-means algorithm. Fig. 7 depicts the sum of SP delay time for various initialization numbers of both approaches. The result shows that the cluster center initialization with careful seeding always guarantees that the packet delivery time between the cluster center and other nodes in the sub-network to be shortened.
Furthermore, we evaluate the end-to-end latency of each application data flow in the network. As depicted in Fig. 8, we simulate experimental comparison between our proposed strategy with recent state-of-the-art solutions for latency minimization problem, those are, 1) the simulated annealing-based method [22], 2) the BSAS-based scheme [26], 3) the MST-based procedure [27], 4) the modified BSAS-based approach [28], and 5) the original K-means algorithm. Moreover, we run all the threshold-based graph clustering methods twice, with and without the careful threshold selection procedure.
The simulation result shows that on both network topologies, our strategy using the optimized K-means clustering  The average latency of application data flows in both FatTree and Abilene network topologies. It is shown that our middlebox placement method based on the optimized K-means clustering algorithm is outperform the recent state-of-the-art approaches, ranging from the simulated annealing-based method to the other graph clustering-based procedures, i.e., the BSAS, the MST, and the original K-means algorithms.
algorithm provides an enormous impact on reducing endto-end flow latency. The average end-to-end flow latency is around 25.22% and 23.81% lower compared with the totally random and the simulated annealing placement methods, respectively. Moreover, the average latency minimization is also improved around 18.44% and 11.49% are compared to other graph clustering-based placement approaches, the BSAS-based scheme, and the MST-based procedure, respectively. Hence, these results prove the intuitive properties and considerations, as described in the previous section. First, with the knowledge of application data services in advance, it is better to put the NIS middleboxes as near as possible to the data flow ingress-switches and divide VOLUME 8, 2020 them into several clusters. Second, the graph clustering-based placement using an arbitrarily clusters' threshold assignment should be avoided. Third, the cluster center initialization method plays a critical role, in which careful seeding initialization procedure proposed in this paper is essential to find all NIS middleboxes placement with minimum latency in SDN/NFV-based EI cyberinfrastructure.
Next, Fig. 9 shows the average of NIS middleboxes resource utilization from all placement strategies. Based on these results, we conclude that the utilization increases along with the increasing number of application data flow. It should be noted that the middlebox processing power capacity depends on the available CPU in each middlebox. Hence, we apply the proposed dynamic resource allocation mechanism for 1) minimizing the NIS processing delay considering the middleboxes processing capacity constraint in (14), and 2) avoiding the functionality failures due to overload usage of reliability-aware implementation as mentioned in the previous section. Figure 10 depicts the effects of the proposed dynamic resource allocation scheme in latency minimization. It is shown that the average end-to-end latency decreases compared to just merely the graph cluster analysis approach. Moreover, compared with the incremental resource allocation approach in [43], our dynamic resource allocation scheme optimizes further latency minimization around 4.24%. The best improvement comes from the fifty percent usage-ratio threshold configuration. However, it should be noted that the prediction error of usage-ratio is still around 17.13%. Hence, a better setting of the regression tree algorithm is needed to be implemented to improve the performance. To this end, enhancement using an ensembling method with other AI-driven predictive algorithms or performing a deep learning analysis can be applied in the future.

C. CHALLENGES AND DISCUSSIONS
The latency minimization analysis demonstrates in the previous subsection prove that our proposed strategy can be utilized to provide the latency-optimal NIS in SDN/NFVbased EI cyberinfrastructure. Furthermore, Table 7 depicts the detailed comparison of our contribution to the existing works in the building of cyberinfrastructure for end-to-end interaction across the entire value chain in the EI using the SDN approach. However, some challenges need to be handled in future research, such as: 1) Since there is no unique criterion to define the structure of EI cyberinfrastructure, the energy stakeholders may implement their own network topology or that suggested by the CSP. Therefore, it will be growing both in size and complexity. Hence, a loop-based topology analysis [40] may be needed to be adopted to provide reliable SDN/NFV-based EI cyberinfrastructure with a more dynamic network topology in the future. 2) To analyze application data flows in this work, a detailed SAM for NIS is required. However, to avoid information leaked by non-trusted parties, it would better to also be provided with a privacy-preserving data scheme, e.g., using the so-called differential privacy mechanism [53]. 3) Even though the objective of our work is to minimize end-to-end latency. However, more targets, such as energy-saving scenarios, could be implemented in the future. Furthermore, effective resource management based on traffic demand, as depicted in [54], may also be adopted.

VI. CONCLUSION
In this paper, the utilization of SDN/NFV-based NIS for fulfilling the cyber-security requirements in the EI ecosystem has been introduced. Furthermore, the AI-powered solution has been proposed to deploy NIS with the minimum endto-end flow latency. This solution consisted of two phases: 1) NIS middlebox placement based on the optimized Kmeans-based graph clustering analysis, 2) dynamic resource allocation using predicted NIS usage-ratio based on the regression tree analysis. Moreover, the evaluation results have verified that our proposed approach could improve latency minimization significantly in two network topologies, i.e., Abilene and FatTree. The average end-to-end latency is more than 15% lower compared to the stateof-the-art threshold-based clustering algorithm. This result proves our intuitive properties and considerations that the graph clustering-based placement using an arbitrarily clusters' threshold assignment should be avoided, and the cluster center initialization method plays a critical role. However, even though the main objective of this paper is minimizing flow latency, more targets such as energy-saving or more complex topology scenarios can be implemented in the future.