An Aggressive Migration Strategy for Service Function Chaining in the Core Cloud

Service Function Chaining (SFC) is regarded as an important concept for next-generation communication networks because it can flexibly tackle diverse usage scenarios. Due to SFC requests’ life-cycle and resource adjustment, the distribution of the remaining physical resources may become unbalanced, which brings negative effects to subsequent SFC requests as well as network operators. In this paper, we investigate the network SFC migration problem in the core cloud under the premise of considering the migration cost and the balance of physical resource distribution. We first model the SFC migration problem as an integer linear program and propose an aggressive migration strategy that can effectively reduce the imbalance of physical resource distribution. Then, we employ two state-of-the-art heuristics to allocate resources for subsequent SFC requests. The simulation results show that migrating SFC requests in the initial service queue can bring favorable feedback to subsequent requests as well as network operators. Compared to the conservative migration strategy, our proposed migration strategy can mitigate the imbalance of physical resource distribution more effectively, and thus the acceptance ratio of subsequent SFC requests, physical resources utilization, and the long-term profit of network operators can be further improved.


I. INTRODUCTION
As technology evolves, a range of new vertical use cases emerges [1], such as online education, virtual reality, and autonomous manufacturing.However, today's mobile communication networks employ a one-size-fits-all approach to providing services, regardless of the diverging requirements of vertical services.In addition, some traditional network functions, such as firewalls, load balancing, and deep packet inspection, are mostly carried by specific physical devices.Therefore, in order to meet the ever-increasing performance demands of users, network operators have to spend expensive capital to maintain old equipment as well as purchase new equipment.Consequently, it is necessary to update the network This research work was supported in part by the Fujian Province Natural Science Foundation of China under Agreement 2020J01574 and the Industry-university-research Innovation Fund of China under Agreement 2021FNA05003.This work was also supported by the CHARITY project that has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 101016509.It is also supported in part by the Academy of Finland 6Genesis project under Grant No. 318927 and IDEA-MILL with grant number 33593.(Corresponding authors: Tarik Taleb and Zhaogang Shu) Haoxian Feng, Zhaogang Shu, Yuantao Wang and Zhiwei Liu are with the computer and information college, Fujian Agriculture and Forestry University, Fuzhou, China.Email: fenghx@fafu.edu.cn;zgshu@fafu.edu.cn;ytwang@fafu.edu.cn;liuzhiwei@fafu.edu.cnTarik Taleb is with the Center of Wireless Communications, The University of Oulu, Finland.Email: tarik.taleb@oulu.fi.
architecture with the ability to address diverse application demands and decrease network operators' operation expenses.
Software Defined Networks (SDN) [2] and Network Function Virtualization (NFV) [3] have emerged as promising approaches to address the above limitations.As SDN decouples the control plane and data plane, it can manage the network in a centralized, flexible way.NFV utilizes virtualization technology to separate specific network functions from dedicated devices to general-purpose commodity hardware.Thus Virtual Network Functions (VNFs) can be placed dynamically at appropriate locations on the network to provide services to the users.Meanwhile, with the rapid development of NFV, VNFs can not only scale dynamically under various traffic conditions to avoid performance degradation and guarantee users' Quality of Experience (QoE) [4], [5], but can be also deployed in clusters to overcome the shortcomings of singlepoint failure and scalability [6].
Based on SDN/NFV technologies, Service Function Chaining (SFC) [7], standardized by the Internet Engineering Task Force (IETF), is regarded as an important networking concept to provide users with flexible services.Typically, SFC comprises a sequence of VNFs and the traffic needs to be steered to traverse these VNFs in a predefined order [8], [9].As it is essential to deploy SFC onto the physical network, at present, there has been some research on the SFC deployment problem [10]- [13] to focus on how to reduce network providers' operational costs, increase resource utilization rate, and guarantee users' Quality of Service (QoS).When a SFC request is received, the management and orchestration (MANO) layer of the network system allocates resources and makes configurations for it, and also takes responsibility for recycling resources after the life-cycle of a request [14].Furthermore, to avoid QoE and Service Level Agreement (SLA) violations caused by traffic load variations [15], [16], MANO should dynamically adjust the allocated resources of SFC requests [17], [18].In addition, to save the allocated resources [19], and adapt to the users' mobility [20], MANO sometimes even has to migrate some VNFs of a request.However, the above adjustment process may result in the unbalanced distribution of physical resources.Different from previous research work [18]- [22], our work mainly answers the following two questions.A: Will uneven distribution of physical resources adversely affect subsequent SFC requests and network operators?B: Can we reduce the potential negative impact of the uneven distribution of physical resources by migrating SFC requests in the initial service queue?
Due to resource adjustment and the departures of some SFC requests, after a period of time, some VNF-enabled nodes and links of the substrate network may be overloaded, which may bring negative effects on the subsequent SFC requests and network operators (e.g., lower request acceptance ratio, lower long-term profit, ect.).Let us use the example in Fig. 1 to illustrate the SFC migration problem in the core cloud.In the initial service queue, there are a total of three SFC requests, and the resource requirements of SFC request 1 and SFC request 2 are small.Since it is easier to find viable candidate resource allocation solutions for requests with small resource requirements in this case, MANO can consider migrating SFC request 1 or SFC request 2. We take the migration of SFC request 2 as an example.As shown in Fig. 1, SFC request 2's transmission path can be migrated from (S1 and its VNF1, VNF2 can be migrated from V2, V4 to V1, V3, respectively.Although SFC migration brings costs, it can make the distribution of physical resources more balanced, which may have a positive feedback on subsequent SFC requests and network operators.In this paper, we investigate the SFC migration problem caused by the imbalanced distribution of physical resources in the core cloud.We first use the standard deviation to represent the physical resource distribution of the substrate network, then, jointly considering the migration cost and some constraints (e.g., resource limitation, latency constraint, ect), we formulate the SFC migration problem as an Integer Linear Programming (ILP) problem.Inspired by some relevant work [20], [22], we design a conservative migration strategy to mitigate the imbalance of physical resource distribution.However, the above-mentioned work focused too much on the migration cost.Consequently, only some VNFs (i.e., sometimes only the last VNF of the SFC) are migrated.Therefore, it may not effectively balance the remaining physical resources.To overcome this shortcoming, we propose an aggressive migration strategy to migrate the whole SFC.After that, we employ two state-of-the-art heuristics methods, BestFit [23], [24], and CN [25] to allocate resources for the subsequent SFC requests, and observe the changes in some metrics, such as the acceptance ratio of subsequent SFC requests and the operator's long-term profit, before and after SFC migration.The main contributions of this paper are summarized as follows: • We formulate the SFC migration problem as an ILP with the aim of minimizing the standard deviation of physical resource distribution as well as the migration cost.• We propose an aggressive SFC migration strategy that prioritizes mitigating imbalances in resource distribution by migrating the total SFC.Then, we employ two stateof-the-art heuristics to allocate resources for the subsequent SFC requests.• Simulation results reveal that our proposed migration strategy can alleviate the imbalance of initial network resource distribution by 14.991% compared with the benchmark strategy.For the different resource allocation heuristics, after SFC migration, the acceptance ratio of subsequent SFC requests increased by 11.75% and 6.84% on average, respectively, and the long-term profit of network operators increased by 9.49% and 7.84% on average, respectively.
The remainder of this paper is organized as follows-Section II reviews some related work.Section III formulates the target problem.Section IV introduces our proposed heuristics.Section V presents the obtained numerical results.Finally, the paper concludes in Section VI.

II. RELATED WORK
Network slicing and SFC migration are two hot research topics [26].In some application scenarios, e.g., automated driving, the location of a user's terminal changes as per the mobility of the user.Taleb et al. [27] introduced the Follow-Me Cloud (FMC) concept and proposed its framework.Then, they applied a Markov Decision Process (MDP) based method to optimize migration decisions [28].Aissioui et al. [29] applied their Follow-Me edge Cloud solution in an automated driving use case.When migrating the VNFs of SFC, operators also need to consider reducing the migration time.Addad et al. carried out related research work [30], [31] on this.For applications depending on synchronization, they designed, proposed and evaluated four SFC migration patterns [30], and they found no clear winner in their presented patterns.Further, they designed a shared file system-based method [31] to decrease both the migration time and the service downtime.
Although some studies aim to support the mobility of services, their ultimate optimization goals are different.In the hybrid environment of the core cloud and edge cloud, Mada et al. [19] formulated the SFC migration problem as an ILP and used the Gurobi optimizer to solve it.In their work, researchers focused on how to minimize the allocated resources to the deployed services but ignored migration cost.Due to the high computation cost of the Gurobi optimizer, their method is infeasible in larger networks.Zhao et al. [20] investigated the problem of SFC migration caused by user movement in cloud-fog computing environments.They also formulated the SFC migration problem as an ILP and proposed two heuristic methods to solve it.Considering migration cost, their methods migrate some VNFs of SFC.However, they ignored the life cycles of SFCs and the distribution of the remaining physical resources.
Some researchers employed Artificial Intelligence (AI) methods to solve the slice and SFC migration problem.Addad et al. [21] designed, modeled, and evaluated two Deep Reinforcement Learning-based algorithms to allocate bandwidth resources for SFCs during migration.Their experiment results showed that a Deep Deterministic Policy Gradient (DDPG) had better performance than a Deep Q-Network (DQN).However, their proposed AI agent can only detect the required bandwidth for a given workflow, and their work did not consider migration cost.Wei et al. [18] investigated the problem of slice reconfiguration caused by fluctuations in user resource requirements in core cloud environments.They formulated the reconfiguration problem as a MDP problem and proposed a discrete Branching Dueling Q-network to solve it.Although their method took the reconfiguration cost into account, their action set was designed for an individual slice, and the trained network model was not universal for SFCs with different resource requirements.For the above DRL-based solutions, once the environment or action set changes, e.g., network topology, SFC requirements, etc., they must retrain the AI agent [32].As the training of deep network models generally requires high computational and time overhead, these methods have certain limitations.In addition, none of the above research work considers the balance of physical resource distribution after migration or reconfiguration.
Few researchers have considered migrating the VNFs running on overloaded nodes.Zhang et al. [22] first proposed an adaptive interference-aware method to allocate resources for slice requests, which can handle VNF interference [33] caused by resource contention.Then, to further improve the long-term total profit of slices and to cope with the migration cost, they proposed a lazy migration strategy that migrates the last VNF of slices to optimize physical resource distribution, so that the network operator can accept slice requests that should be rejected.Although their method can alleviate the burden of high-load nodes to a certain extent, it does not consider the balance of resource distribution from a global perspective.
At present, most of the research on slice and SFC migration is related to supporting users' mobility, and a small part [22] to optimizing global resources.However, to the best of our knowledge, there is no prior work that considers the impact of initial physical resource distribution on subsequent SFC requests and network operators' long-term profit, which is the focus of our work in this paper.A comparison of related works is given in Table ??.

III. PROBLEM STATEMENT AND FORMULATION
In this paper, we study the SFC migration problem for SFC requests in the initial service queue.In a physical network, operators can allocate resources for users based on their specific QoS requirements.The QoS requirements and lifecycles of different requests are diverse.Furthermore, during their life cycles, MANO may dynamically adjust the allocated resources according to changes in demand.The above factors cause the resources allocation solutions for the SFC requests in the service queue to become unstable, resulting in the uneven distribution of physical resources.
In this paper, we consider the following scenario.At the initial moment, there are some permanent SFC requests in the service queue, these requests occupy more physical resources than normal requests, and the allocation solutions of their resources are random.MANO needs to migrate these requests under the premise of some constraints (e.g., the association of VNF instances, bandwidth demand, etc.) to alleviate the uneven distribution of physical resources.
Since the essence of the SFC migration problem is to calculate new resource allocation solutions for the requests in the service queue, we first describe the substrate network and SFC request models and then propose a mathematical formulation of the SFC migration problem.For ease of reference, the notations used in this paper are summarized in Table II.

A. Substrate Network
Similar to some related work, we define the substrate network as a weighted directed graph G = (N, E), where N and E denote the sets of physical nodes and links, respectively.Typically, a substrate network is composed of VNF-enabled nodes N V and common nodes N F , so the set of physical nodes can be further denoted as N = N V ∪ N F .VNF-enabled nodes can provide certain types of VNFs, such as Network Address Translation (NAT) and Firewall; while common nodes can be only used for packet forwarding.Note that the forwarding function can be seen as a special network function [18] and its resource consumption is relatively small, so the resource consumption caused by the forwarding function is ignored in our work.
We indicate the total computational resource capacity of the VNF-enabled node i ∈ N V as C i , while other resources (e.g., memory resources, storage resources) are sufficient.Each physical link (i, j) ∈ E has the limited bandwidth of B i,j , and the propagation delay of a physical link (i, j) is denoted as τ i,j .

B. SFC Request
In this paper, we define a SFC request r as a linear chain, and it can be denoted as r = (s r , d r , F r , τ r , c r , b r ),  where s r , d r denote the source and destination of r, F r denotes the SFC requirement, and τ r , c r , b r denote the latency requirement, the CPU resource requirement for VNFs, and the bandwidth resource requirement, respectively.For example, in Fig. 1, SFC request 3 can be denoted as r 3 = (S 2 , S 6 , {VNF1, VNF2, VNF3}, τ 3 , c 3 , b 3 ).

C. Problem Formulation
Now we formally propose the mathematical formulation of the SFC migration problem.We start with the constraints that MANO should obey when recomputing the resource allocation solutions for SFC requests in the service queue.
We first model the VNFs mapping SFC requests, and define some binary variables.h i (π) indicates whether or not node i is capable of VNF π.X i,r (π r m ) indicates whether or not VNF π r m is placed onto physical node i, where π r m represents the m th VNF of SFC request r.To ensure VNFs are placed onto the VNF-enabled physical nodes, we have: Referring to some relevant works [20], [22], VNFs are not allowed to be split in our current work, so each VNF of a SFC request can be only placed onto one VNF-enabled physical node.Thus, we have the following constraint: In addition, we also require that each VNF-enabled physical node provides at most one VNF for each F r , and this can be ensured if Eq. (3) holds.
Then we consider the virtual link mapping of SFC requests.We do not allow path splitting for the purpose of avoiding coordination overhead.Similar to [20], we denote P r as the transmission path of request r, and M r as the VNFs placement solution of F r .Each transmission path should obey the following constraints.
Constraint ( 4) is latency related.The total delay of the transmission path should be less than the delay required by the request.Constraint (5) requires that the transmission path must pass through the physical nodes that provide the VNF service for request r.
Finally, we consider the constraints of the resource upper bounds.Similarly, we define a binary variable Z i,j (r, π r m ) to indicate whether (r, π r m ) is mapped on a physical link (i, j) or not, where (r, π r m ) represents the virtual link between virtual node π r m and π r m+1 .Specially, (r, π r 0 ) represents the virtual link from s r to the first VNF, and (r, π r Lr ) represents the virtual link from the last VNF to d r .As mentioned above, we ignore the resource consumption of the forwarding function, so the total computational resource consumed on a VNF-enabled physical node i and the total bandwidth resource consumed on a physical link (i, j) are: and respectively, where c r represents the CPU resource requirement for a single VNF of request r, and b r represents the bandwidth resource requirement of request r.
Similar to [10], [34], we assume that within a request r, the computational resource requirements of VNFs are the same.Thus, we have the following capacity constraints for VNFenabled physical nodes and physical links.
Now, let us use the standard deviation σ G to measure the distribution of physical resources.σ G can be calculated as: where N U M (N V ) and N U M (E) represent the number of VNF-enabled physical nodes and physical links in the substrate network G, respectively.R C and R B represent the average value of the consumed resource in N V and E, respectively.In Fig. 1, after migration, since the overloaded nodes and links no longer exist in the substrate network, σ G can be reduced.
During SFC migration, we also have to consider the migration cost.Referring to some related works [18], [20], [22], we introduce the migration cost o r of a SFC request r to represent the extra cost of continuing processing the total amount of the unprocessed packets of r.We use t r to indicate the time when the migration of r starts, t ′ r to indicate the time when the migration of r ends and u r (t) to represent the flow rate of r at time t.Thus, during migration, the total amount of the unprocessed packets of r can be calculated as Referring to the definition of migration cost in [22], we also use the bandwidth resource requirement of r to represent u r (t).Since the flow of r should be orderly-executed based on a set of VNFs, the further up in the SFC the VNFs we intend to migrate are, the larger the effects they will bring to the whole SFC [22].We use Len(P o r , P m r ) to denote how many new virtual links of unprocessed packets of r should be transmitted after migration.For example, in Fig. 1, the original transmission path of SFC request 2

and its transmission path after migration
).In such case, Len(P o 2 , P m 2 ) = 4, because after virtual node V1, subsequent links begin to change.Therefore, we define o r as: In addition, to calculate the total migration cost O G , we need to accumulate the migration cost of each SFC request in the service queue, and the total migration cost can be represented as: Finally, taking the distribution of physical resources as well as the migration cost into account, we define the SFC migration problem as:

IV. HEURISTICS FOR SFC MIGRATION
In this section, we elaborate on the conservative migration strategy inspired by [20], [22] and our proposed aggressive migration strategy, respectively.

A. A conservative migration strategy
To quickly restore the user's service or reduce the effect caused by migration on the whole SFC, some researchers [20], [22] are prone to migrate only the last VNF of the SFC.Inspired by this idea, we designed a conservative migration strategy that is performed through the following steps.
Step 1: fetch request with low resource requirement.The smaller the resource requirement of a request, the more candidate resource allocation solutions there will be.We calculate the resource requirement of a request through Eq. ( 14) (In this paper, we prioritize computational resources due to their higher profit).Before migration, we sort all requests in SQ by their Y r and select the request with the smallest Y r for migration.
Step 2: resource recovery.Before SFC migration, MANO should recover the physical resources that have been allocated to request r, including computational resources in the VNFenabled physical node and bandwidth resources in the physical link.
Step 3: determine the stable section.Since the conservative migration strategy only migrates the last VNF of the SFC, MANO needs to determine the stable section of the original resource allocation solution.We use m s r , p s r to represent the stable section of M o r and the stable section of P o r , respectively.Then we denote L r as the length of F r , m r Lr as the last element of M o r , and denote m r pen as the penultimate element of M o r .Clearly, m s r equals to M r − m r Lr , and p s r is (s r → ... → m r pen ).
Step 4: recalculate the resource solution for the migration section.This step needs to solve the following two problems.(a) Where the last VNF of F r should be migrated to.(b) Which transmission path from m r pen to d r should be chosen.We use π r Lr to represent the last VNF of F r , m re r , p re r to represent the new resource allocation solution for π r Lr , and the new allocation solution for the transmission path from m r pen to d r , respectively.To avoid loops, we first need to remove elements of p s r from G (except for m r pen for the purpose of path calculation 1 ) and get the pruned substrate network G p1 .Then, with the constraints of (1), (3), we calculate the candidate physical node set N r for π r Lr .Typically, , where N ′ V represent the set of VNF-enabled physical nodes in G p1 .Then, we select the physical node i most that not only can provide service for π r Lr , but also with the most remaining computational resource in N r , and place π r Lr on i most .Next, we remove the physical links that cannot meet the bandwidth requirement of r in G p1 , and get another pruned substrate network G p2 .We can get all paths from m pen to d r in G p2 with a modified Depth First Search (DFS) method [35].Then we select the paths that obey constraint (4), (5) to form the set P r of feasible solutions to p re r , and place the virtual path from m r pen to d r on the path p least with the least standard deviation of the remaining bandwidth resources in P r .After the above process, we get m re r equal to i most and p re r equal to p least .
Step 5: resource remapping.Once MANO gets p s r , p re r , m s r , m re r , it can determine a new resources allocation solution for request r.Then, MANO should remap the resource for request r based on P m r , M m r .Algorithm 1 shows the pseudo-code of the conservative migration strategy.The Resources Recovery (RR) procedure is used to carry out step 2, and the Recalculate Resource Solution (RRS) procedure is used to carry out step 4. Step 5 is carried out through line 8 to line 13.
In Fig. 1, since the resource requirements of SFC request 1 and 2 are relatively small, MANO will migrate them first.Here, we use the migration process of SFC request 2 to illustrate the conservative migration strategy.Firstly, MANO needs to recover the physical resources previously allocated to SFC request 2, as shown in Fig. 2(a).Secondly, MANO needs to determine the stable section of SFC request 2's original resource allocation solution, as shown in Fig. 2(b), where m s r is (S1 → V1 → V2), and p s r is {V2}.Thirdly, MANO has to determine where the last VNF (VNF2) of F 2 should be migrated to, and which transmission path from m 2 pen to d 2 should be chosen.To solve the above problems, MANO first removes some nodes and links in the substrate network to avoid loops, as shown in Fig. 2(c).Then, since V3 has the most remaining computing resources, MANO maps VNF2 on V3 when constraint ( 1) is met and sets m re 2 to V3, as shown in Fig. 2(d).After that, MANO calculates feasible paths from m 2 pen to d 2 in the pruned substrate network.Under constraints (4), ( 5 computation to calculate all the paths from node s to node d in graph G, where P, E, N denote the set of candidate paths from s to d, the set of physical links in G, and the set of physical nodes in G, respectively.In our work, we let E r , N r to represent the physical links set and physical nodes sets in each G p2 , and we let

B. An aggressive migration strategy
In order to reduce the migration cost, the conservative migration strategy only migrates the last VNF of the SFC, and this will cause overloaded nodes and links in the substrate Call RRS procedure, then get p re r , m re r ; 7: Let M m r = m s r ∪ m re r , P m r = p s r ∪ p re r ; 8: for each node i ∈ M m r do 9: end for 11: for each link (i, j) ∈ P m r do 12: B i,j = B i,j − b r ; 13: end for 14: Calculate migration cost o r of request r according to Eq.( 11); 15: end if 10: end for 11: return G rec .network to still exist after SFC migration, as shown in Fig. 2(e).To distribute physical resources more evenly, we propose an aggressive migration strategy that migrates the total SFC, which is performed through the following steps.
Step 1, Step 2 are the same as Step 1 and Step 2 of conservative migration strategy.
Step 3:calculate a new resources allocation solution.To solve the SFC migration problem, we split it into VNF migration and virtual link migration.For VNF migration, we greedily place VNFs on the physical nodes with the most remaining resources.For virtual link migration, we first remove the physical links that cannot meet the bandwidth requirement of r and get a pruned network G p1 , and then Remove i and its directly connected links form G; 4: end for 5: Remove the physical links that can not meet the bandwidth requirement of r. if (p ∪ p s r ) allows constraints ( 4),( 5) then 10: end if 12: end for 13: Select the path p with the smallest standard deviation of the remaining bandwidth resources from P r , then let p m r = p; 14: return p re r , m re r .
we get all paths from s r to d r in G p1 through a modified DFS method and select the paths that obey constraints ( 4), (5) to form the set P r of feasible solutions to P m r .Finally, we place P m r on the path p with the smallest standard deviation of the remaining bandwidth resources in P r .
Step 4: resources remapping.MANO needs to remap the resources based on the new resources allocation solution calculated by step 3.
The procedure of aggressive migration is shown in Alg. 2. Line 5 to line 16 are used to calculate new resources allocation solutions P m r , M m r for request r.And step 4 is carried out through line 17 to line 22.Similarly, we take the migration process of SFC request 2 in Fig. 1 as an example and the process shown in Fig. 3. Fig. 3(a) corresponds with the process of resource recovery, Fig. 3(b) corresponds with the process of VNF migration, and Fig. 3(c) corresponds with the process of virtual link migration and resource remapping.
Similarly, we take the migration process of SFC request 2 in Fig. 1 as an example.As shown in Fig. 3(a), MANO should also recover the physical computational and bandwidth resources previously allocated to SFC request 2.Then, MANO greedily places the VNFs of F 2 on the physical nodes with the most remaining resources, and sets M m 2 to {V1, V2}, as shown in Fig. 3(b).After that, MANO first removes the physical links that cannot meet the bandwidth requirement of SFC request 2 in the substrate network, and then it calculates the feasible paths from s 2 to d 2 in the pruned substrate network under constraints (4), (5).Since the standard deviation of the bandwidth resource distribution of path (S1 → V1 → V3 → S4 → S6 → S5) is the smallest, MANO sets P m 2 to it.Finally, MANO reallocates the resource for SFC request 2 based on M m 2 , P m 2 , and the migration effect is shown in Fig. 3(c).Note that, after aggressive migration, overloaded nodes and links no longer exist in the substrate network, so the aggressive migration strategy is expected to make the distribution of physical resources more balanced.
Likewise, we analyze the time complexity of our proposed aggressive migration strategy.Similarly, the computation required for sorting all requests Note that, before running the modified DFS method, the conservative migration strategy removes more nodes and links from the substrate network, so its B con , E con , and N con are smaller than those of the aggressive migration strategy.Therefore, the time complexity of the conservative migration strategy is slightly smaller than that of the aggressive migration strategy.

V. NUMERICAL RESULTS
In this section, we demonstrate the performance evaluation of our proposed migration strategy.We first discuss the simulation setup used to evaluate the algorithms in our work.We then introduce the state-of-the-art heuristics for allocating resources for subsequent SFC requests.Finally, we compare our aggressive migration strategy with the conservative migration strategy and describe our main simulation results and analysis.

A. Simulation settings
In our simulations, we construct a substrate network as is shown in Fig. 4, which is widely used in network slicing and SFC research, such as [17], [18].The parameters of the substrate network are listed in Table III.In the substrate network, there are 7 VNF-enabled physical nodes, and they can provide all kinds of VNF services.Currently, we do not consider the difference in the resource upper limit between different VNF-enabled physical nodes and physical links.Therefore, the computational resources of VNF-enabled physical nodes are set to 10000 units.As some researchers assume that processing one unit of flow requires one unit of computational capacity [38], we set the bandwidth resources of physical links to 10000 units too.In addition, the latency of physical links was set to 1ms.
The parameters of various SFC requests in our work are shown in Table III.In real-world scenarios, there will be permanent and temporary network service requests [36], [37].Therefore, we designed a type of permanent request, which  Select r with the minimal Y r , call RR procedure, then let G = G rec , M m r ={}, candidate VNF-EN node set N r = N V , candidate transmission path P r ={}; 5: Select the node i most with the most remaining resources from N ; 7: end for Copy G, get duplicate G p1 and remove the physical links that can not meet the bandwidth requirement of request r in G p1 ; 10: Run modified Depth First Search with s r , d r , G p1 , then generate a set of candidate paths P c r ; 11: for each path p ∈ P c r do 12: if p allows constraints (4), (5) then 13: P r = P r + p; Select the path p with the smallest standard deviation of the remaining bandwidth resources from P r , then let P m r = p; 17: for each link (i, j) ∈ P m r do 18: end for 20: for each node i ∈ M m r do 21: end for

23:
Calculate migration cost o r of request r according to Eq.( O G = O G + o r , SQ = SQ − r, and then migrate the next SFC request; 25: end while 26: return G ′ , O G . will occupy a large amount of physical resources, but correspondingly, their number was set to be scarce.The other two types of requests are temporary requests.They take up fewer physical resources than permanent requests, but their number is very large.What is more, the bandwidth resource requirement of request r is related to its computational resource requirement, so we get b r = β • c r .As we also assume that processing one unit of data flow requires one unit of computational capacity [38], we set β = 1 in our current work.The source and destination of all SFC requests are randomly generated from the S and D nodes in the substrate network.
We assume that at the initial moment, there are 6 permanent  requests in the service queue.Under the premise of satisfying resource mapping constraints, the resource allocation solutions of these requests are random, so as to simulate the randomness of the initial physical resources distribution.After the initial moment, we assume that the subsequent requests consist of Type 2 and Type 3 requests randomly, and these SFC requests are issued following a Poisson process of rate λ, where λ is set to 5 in our work.The migration strategies are evaluated using the Python programming language and an extended package for graph theory called networkx [39].

B. Heuristics for subsequent SFC requests
The BestFit [23], [24] and CN [25] heuristics are employed for allocating resources for the subsequent SFC requests, and below we give a brief description of them.
• BestFit: First, the physical node with the most remaining computational resources is greedily selected to place the VNFs requested by a request, and then the shortest path between these nodes is calculated to connect these VNFs in series under the premise of meeting the bandwidth requirements.
• CN: CN similarly divides resource allocating into two stages.First, it calculates the importance of each node according to the degree, betweenness centrality of the nodes, the remaining computational resources of the nodes, and the remaining bandwidth resources of the links to which the nodes are directly connected.Then, the VNFs requested by SFC requests are greedily placed on physical nodes with high importance.Finally, under the premise of meeting the bandwidth requirements, the shortest path between these physical nodes is calculated to connect the VNFs in series.

C. Evaluation indicators
• Standard deviation of physical resources distribution: The standard deviation of physical resource distribution is used to measure the balance of physical resource distribution, which can be calculated by Eq. ( 10).
• Total migration cost: The total migration cost is used to measure the total cost of migrating the SFC requests in the service queue, which can be calculated by Eq. ( 12).• Acceptance ratio: The SFC request acceptance ratio at time slot T can be defined as: where N U M N SR S and N M N SR the number of SFC requests that are successfully allocated resources and the number of total SFC • Node The utilization of the computational resource of VNF-enabled physical nodes at time slot T can be defined as: • Link utilization: The utilization of the bandwidth resource of physical links at time slot T can be defined as: • Long-term profit: The total profit earned by the network operator for serving SFC requests from the initial time to time slot T can be defined as: where W cpu and W bw represent the benefit of per computational unit and per bandwidth unit, respectively, and P V r denotes the virtual path of request r.In our work, W cpu , and W bw are set to 0.1, and 0.05, respectively.

D. Simulation results and analysis
We first compare the changes in resource distribution in the substrate network after two different SFC migration strategies.As shown in Fig. 5, in a single experimental instance, our proposed migration strategy can make the initial physical resource distribution more balanced.We then randomly generate 20 other instances of the problem (13).Fig. 6 demonstrates the effectiveness of our proposed strategy and compared with the conservative migration strategy, the σ G of our proposed strategy is 14.991% lower on average.However, as our proposed strategy migrates the total SFC, it causes more migration cost, as shown in Fig. 7.In 20 experimental instances, the migration cost of our proposed strategy is on average 25.5% higher than with the conservative migration, as shown in Fig. 8.The above results are summarized in Table V.   Next, we compare the changes in the acceptance ratio of subsequent SFC requests, the utilization of physical resources, and the long-term total profit, before and after SFC migration.
We first analyze the simulation results for a single experimental instance.As shown in Fig. 9, and 10, it can be seen that as time goes by, MANO receives more and more SFC requests, and the physical resources become more and more strained.Therefore, after a period of time, the acceptance ratio of SFC requests starts to decrease.By migrating SFC requests in the service queue at the initial moment through our proposed strategy, we can make the operators receive more SFC requests, and thus increase the final acceptance ratio.However, the conservative migration strategy has a negative impact on the final acceptance ratio.We then analyze changes in physical resource utilization.As shown in Fig. 11, 12, and 13, for both "BestFit" and "CN" heuristics, our proposed migration strategy improves final resource utilization and longterm profit.The "BestFit" heuristic, although the conservative migration strategy reduces the acceptance ratio of SFC requests, improves the final resource utilization.For the "CN" heuristic, the conservative migration strategy reduces both the acceptance ratio of SFC requests and the final resource utilization.Interestingly, however, the conservative migration strategy advances the time when the physical resource utilization reaches a plateau.From time slot 30 to time slot 44, the resource utilization of the "Conservative Migration + CN" strategy is higher than the pure "CN" strategy.Although the resource utilization of the pure "CN" strategy exceeds that of the "Conservative Migration + CN" strategy from time slot 45, it still needs a long period of time for the pure "CN" strategy to catch up with the long-term profit of the "Conservative Migration + CN" strategy.From time slot 45 to time slot 60, the gap between the pure "CN" strategy and the "Conservative Migration + CN" strategy in long-term profit gradually narrows, but until time slot 60, the long-term profit of the "Conservative Migration" strategy is still higher than that of the pure "CN" strategy.
According to Eq. ( 18), the long-term total profit defined in our work is an obvious time-dependent monotonically increasing function (unless there are no SFC services in the service queue for a certain amount of time, which can be ruled out in our work as there are some permanent SFC requests).Therefore, if MANO no longer receives new SFC requests and no SFC requests leave the service queue, the long-term profits of different schemes will show perfect linear growth with the time slots.In in Fig. 13, the slopes of some straight lines change over time.For example, from time slot 40 to time slot 46, the long-term profit of "Conservative Migration + CN" is actually higher than the long-term profit of "Conservative Migration + BestFit".But after time slot 37, as shown in Fig. 11 and Fig. 12, both the node resources consumption and link resources consumption of "Conservative Migration + BestFit" are higher than that of "Conservative Migration + CN", which results in the long-term profit of "Conservative Migration + BestFit" surpassing the long-term profit of "Conservative Migration + CN" after time slot 49.We then analyze the simulation results for the other 20 experimental instances at time slot 60.As shown in Fig. 14, our proposed aggressive migration strategy improves the acceptance ratio of subsequent SFC requests in most cases (except for instances 4, 5, and 7).Sometimes, MANO may Fig. 12. Different node resources utilization for an experimental instance Fig. 13.Different long-term profit for an experimental instance receive a Type 3 SFC request to earn more profit, and in this case it would reject multiple Type 2 SFC requests.Therefore, it is not weird that in a few cases, the acceptance ratio drops after migration.For "BestFit", the average improvement is 11.75%, and for "CN", the average improvement is 6.84%.As shown in Fig. 16, and 17, for all experimental instances, the aggressive migration strategy improves node resource utilization and long-term profit.For "BestFit", node utilization and long-term profit are increased by an average of 8.14% and 9.49%, respectively.For "CN", node utilization and longterm profit are increased by an average of 4.65% and 7.14%, respectively.In some cases (e.g., instance 3), the aggressive migration strategy reduces link resource utilization, but this does not negatively impact long-term profit because no matter how many physical links a virtual link is mapped to, only one virtual link is charged.
Likewise, for all experimental instances, the conservative migration strategy can also improve the operator's long-term profit, but the improvement is smaller than that of aggressive migration strategies.For "BestFit", the average improvement is 3.05%, and for "CN", the average improvement is 4.62%.Similar to the aggressive migration strategy, sometimes the conservative migration strategy reduces the acceptance ratio of SFC requests and physical resource utilization, but still improves the final long-term profit.Overall, with "BestFit", the conservative migration strategy improves the acceptance ratio of SFC requests, node resource utilization, and link resource utilization by 1.82%, 1.59%, and 1.11% on average, respectively.With "CN", the conservative migration strategy improves the acceptance ratio of SFC requests, node resource utilization, and link resource utilization by 2.54%, 1.93%,  Compared with the conservative migration strategy, the improvements in the evaluation indicators of our proposed migration strategy are better.For "BestFit", the improvements in Fig. 17.Long-term total profit for 20 experimental instances average acceptance ratio for subsequent SFC requests, average node resources utilization, average link resources utilization, and average long-term profit are 9.93%, 6.55%, 3.2%, 6.44% better than those for the conservative migration strategy.For "CN", these numbers are 4.3%, 2.72%, 1.09%, and 2.52%, respectively.In addition, as shown in Fig. 16, in most cases, node resources utilization rates are close to 100% (except for instance 5, 7) after aggressive migration.Though the migration cost of our proposed strategy is on average 25.5% higher, the migration cost is much smaller than the final long-term profit.For "CN", the average migration costs caused by the conservative migration strategy and the aggressive migration strategy are only 0.7074% and 0.8844% of the original final long-term profit.For "BestFit", those numbers are 0.7008% and 0.8795%, respectively.These migration costs are almost negligible compared to the improvement in the final long-term profit.

VI. CONCLUSION AND FUTURE WORK
This paper investigated the impact of the uneven distribution of substrate network resources on network operators and subsequent SFC requests.We first modeled the SFC migration problem as an ILP, then, inspired by some outstanding relevant works [20], [22], we designed a conservative migration strategy and further proposed an aggressive migration strategy to reduce the imbalance of network resources distribution.After migration, we employed two state-of-the-art heuristics to allocate resources for the subsequent SFC requests.The simulation results not only demonstrated that the unbalanced distribution of physical resources indeed affects both the subsequent SFC requests and the network operators negatively, but also showed the efficiency of our proposed strategy.
Yet, we still have to conduct further research to explore the migration problem.Here, we list some aspects we need to improve in the future.

A. More complex application scenarios
At present, we investigated the SFC migration problem in the core cloud.But for some more complex application scenarios, like network slices requests [40], both the edge cloud servers and the core cloud servers should be considered, as they can form complex graphs of VNFs with different service chains.In such cases, the model we defined in current work is not suitable to meet realistic situations.Therefore, we should rethink some of the constraints and design a more complicated model.

B. Further exploration of the optimization goal
As shown in Fig. 17, although in most cases eliminating the unbalanced physical resources distribution does bring positive effect on the network operators, in a small number of instances (e.g., instance 7), the improvement is not obvious.In addition, our current work does not consider the difference in the resource upper limit between different physical nodes, physical links and the potential effects that the topology structure may cause to our optimization goal.Therefore, to study the specific relationship between the unbalanced distribution of physical resources and the long-term profit of network operators, extensive experiments should be carried out in the future.At that time, we will also consider some interesting attributes of the physical nodes and links, such as the degree of a node and the betweenness centrality, to further adjust our optimization goal.

C. Migration research with prediction mechanisms
In this paper, we do not fully take the volatility of services' resource requirements into account.To further guarantee services' QoE or utilize the physical resources, an effective prediction mechanism is required.However, predicting various kinds of services' resource requirements is particularly challenging, because their traffic characteristics are different.To better carry out migration research, we will explore how to design a general artificial intelligence model to predict the resource requirements of various differentiated services in the future.

Fig. 1 .
Fig. 1.Illustration of network SFC migration ), since the standard deviation of the bandwidth resource distribution of path (V2 → V3 → S4 → S6 → S5) is the smallest, MANO maps p re 2 on it.Finally, MANO reallocates the resource for SFC request 2 based on m s 2 ∪ m re 2 , p s 2 ∪ p re 2 , and the migration effect is shown in Fig. 2(e).Next, we analyze the complexity of the conservative migration strategy.First, sorting all requests (line 2) requires O(|SQ| log 2 |SQ|) computation.Then, the first while-loop (line 3) terminates in |SQ| iterations.For the RR procedure, it needs O(|E| + |N V |) computation.For the RRS procedure, its first for-loop (line 2) terminates in |p s r | − 1 iterations, and its second for-loop (line 8) terminates in |P c r | iterations.In addition, for the modified DFS method, a simple path can be found in O(|E| + |N |) [35], which thus causes O(|P | (|E| + |N |)

Procedure 2 V ∩ m s r ); 2 :
Recalculate Resource Solution (RRS) Input: G, b r , c r , p s r , m s r , d r .Output: p m re , m re r .1: Initialization: let s ′ r equal the last element of p s r , candidate transmission path P r ={}, N r = N ′ V − (N ′ for each node i ∈ (p s r − s ′ r ) do 3: (line 2) is O(|SQ| log 2 |SQ|), the first while-loop (line 3) terminates in |SQ| iterations, and the RR procedure needs O(|E| + |N V |) computation.The first (line 5), second (line 11), third (line 17) and fourth (line 20) for-loops need to terminate in |F r |, |P c r |, |P m r |, and |M c r | iterations, respectively.Then, we also let E r , N r to represent the physical links set and physical nodes set in each G p1 , and let F = max r=1 |F r |, E agg = max r=1 |E r |, N agg = max r=1 |N r |, B agg = max r=1 |P c r |.Thus, the time complexity of the aggressive migration strategy is O(|SQ| log 2 |SQ| + |SQ|

Fig. 7 .
Fig. 7. Different migration cost for an experimental instance

Fig. 9 .Fig. 11 .
Fig. 9. Number of SFC requests in the service queue over time

TABLE II SUMMARY
OF NOTATIONS USED.
Notation DefinitionN, E physical nodes (including forwarding nodes N F and VNF-enabled nodes N V ) and links respectivelyτ i,j , Bi,j transmission delay, bandwidth resources of link (i, j) respectively C i computational resources of a VNF-enabled node i sr, dr, τr the source node, destination node, and delay requirement of SFC request r, respectively cr, br CPU resource requirement for a single VNF, and bandwidth resource requirement of SFC request r SQ service queue Fr the set of VNFs of SFC request r π r m the m th VNF of SFC request r Pr the transmission path of SFC request r Mr the VNFs placement solution of Fr (r, π r m ) the virtual link between virtual node π r m and π r m+1 h i (π) binary variable, indicates whether or not node i is capable of VNF π X i,r (π r m ) binary variable, indicates whether or not VNF π r m is placed onto physical node i Z i,j (r, π r m ) binary variable, indicates whether or not virtual link (r, π r m ) is mapped onto physical link (i, j) β unit bandwidth consumption of per computational unit Wcpu, W bw benefit of per bandwidth unit and per computational unit, respectively The last two for-loops (line 8 and line 11) in the conservative migration strategy need to terminate in |M m r |, |P m r | iterations, respectively.Since B con (E con + N con ) is greater than log 2 |SQ|, A con , (|E|+|N |), |M m r |, and |P m r |, the overall time complexity of conservative migration strategy is and then migrate the next SFC request; 16: end while 17: return G ′ , O G .
1: for each link (i, j) ∈ E do 6: Select the node i most with the most remaining resources in N r , let m re r = i most ; 7: Run modified Depth First Search with s Algorithm 2 Aggressive Migration Algorithm Input: initial substrate network G, information of the requests in service queue SQ, including c r , b r , F r , P o r , M o r , s r , d r , τ r .Output: substrate network G ′ after SFC migration, and migration cost O G .1: Initialization: let O G = 0; 2: Sort all r ∈ SQ by its Y r in ascending order; 3: while SQ ̸ = ∅ do

TABLE V AVERAGE
STD AND MIGRATION COST OF 20 EXPERIMENTAL INSTANCES

TABLE VI AVERAGE
RESULTS AND IMPROVEMENT OF 20 EXPERIMENTAL INSTANCES