A Dynamic Virtual Machine Resource Consolidation Strategy Based on a Gray Model and Improved Discrete Particle Swarm Optimization

With the continuous expansion of the scale of cloud datacenters, high energy consumption has become a problem. Virtual machine (VM) consolidation is an effective method of energy management, but overly aggressive consolidation has caused issues such as high service level agreement violation (SLA) rates and excessive migrations. To reduce energy consumption while ensuring the quality of service of a datacenter, a VM dynamic consolidation strategy based on a gray model and an improved discrete particle swarm algorithm (GM-DPSO) is proposed. This strategy uses a gray prediction algorithm for load detection and adjusts the size of the threshold according to the load condition. Then, we select VMs for overloaded and underloaded hosts and complete the placement of VMs through a VM placement strategy based on an improved discrete particle swarm algorithm. This method aims to improve the load balance, establish a mapping between VMs and hosts, use particles to search for the best VM placement in the global scope, and reduce the probability of SLAV. Our algorithm considers the impact of VM migration on system energy consumption and service quality and takes sufficient measures to reduce the number of migrations. To verify the effectiveness and practicality of this strategy, experiments were conducted on actual workloads, and the results were compared with those of other strategies. The experimental results show that the GM-DPSO greatly improves service quality while reducing energy consumption and SLAV by 34.53% and 97.53%, respectively. The methods and theories in this paper provide strong theoretical and practical engineering guidance for large-scale cloud deployment.


I. INTRODUCTION
With the continuous development of cloud computing technology, the scale of cloud datacenters is expanding. Around the globe, 80% of the companies that deploy OpenStack cloud platforms have 100 or more cloud computing servers. The construction of cloud datacenters has increased energy consumption, and the insufficient utilization of cloud servers has further increased the operating costs. Therefore, the efficient The associate editor coordinating the review of this manuscript and approving it for publication was Abdullah Iliyasu . use of datacenters has become an inevitable issue in the era of green and intelligent power consumption. According to statistics, datacenters that do not adopt effective energy-saving measures in China cover an area of 1,000 square meters, and they consume more than 500 kWh of electricity each year. Today, the CPU utilization of most resource center servers is less than 30%, and this usage increases energy consumption and resource waste to some degree [1]. In today's datacenters, cloud providers need to reduce energy consumption to decrease operating costs, and one of the most effective ways is to formulate a virtual machine (VM) consolidation VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ strategy to improve the utilization of dynamic resources [2]- [6]. However, the consolidation of VMs will inevitably lead to a decline in service quality, resulting in Service Level Agreement (SLA) violations. Therefore, when formulating a consolidation strategy, it is necessary to ensure excellent user service quality [7]- [10] while minimizing energy consumption. VM consolidation is a dynamic method that decreases the number of hosts operating in a datacenter through VM migrations to gain the full benefit of computing resources and reduce energy consumption [11], [12]. Traditional VM consolidation methods tend to focus on energy consumption [13], [14] and use aggressive consolidation methods to shut down hosts, but this approach can seriously affect the operating performance of the system. With the continuous development of cloud computing technology in everyday applications, users are becoming more concerned with the stability of the cloud platform service quality. Generally, declines in service quality are mainly due to host overload and excessive VM migrations [10]. Therefore, we consider the service quality of a datacenter from the following perspectives: 1. Reasonable consolidation timing. The consolidation timing needs to be adapted to the utilization ability of the host, and a reasonable consolidation timing can improve the overall performance of the system under different load conditions [8]; 2. Decrease the number of VM migrations [16], [17]. A decrease in the number of VM migrations can significantly mitigate service quality degradation caused by migration; 3. Consider the load balance when placing VMs. Load balancing can balance the remaining resource space of each node, improve the scalability of the system, and reduce the probability of host overload.
The timing of the consolidation of VMs is determined by the load of the host. The load of the host often changes dynamically; if load changes were not considered, invalid migration could occur. In this paper, a gray prediction algorithm based on residual correction is used for load prediction. This algorithm can establish a nonlinear prediction model based on historical load data [19], and the prediction accuracy is higher than that of linear models [8], [20]. Traditional consolidation algorithms [6], [15] are often based on greedy models, and the algorithm complexity is low; however, it is difficult to obtain the global optimal solution with these models. The particle swarm algorithm is a metaheuristic algorithm that can search for a solution that meets the requirements in the global scope and optimizes some aspects of the placement strategy approach [21]. The particle swarm algorithm is usually used to solve continuous problems, so it needs to be coded to solve discrete problems. In this article, we will improve the particle swarm algorithm to adapt to the VM consolidation problem.
According to the above analysis, this paper proposes a VM dynamic consolidation strategy called GM-DPSO based on a gray model and improved discrete particle swarm optimization. We predict the future CPU utilization of the host by using the gray model to adaptively determine the consolidation time. During the process of VM consolidation, we select destination hosts through improved discrete particle swarm optimization to improve load balancing among hosts. The algorithm searches for the optimal placement strategy based on the changes in particle speed and position to avoid a result that becomes trapped in a local optimal solution. The work performed in this article is as follows: 1) We use dual thresholds for load detection and introduce gray prediction and dynamic threshold adjustment to optimize host overload detection and reduce the migration of VMs. 2) A VM selection strategy based on the migration time and CPU utilization is proposed to decrease the migration time and frequency. 3) A VM placement strategy based on an improved discrete particle swarm optimization is proposed to balance the placement of migrated VMs and achieve overall service optimization. 4) We evaluate the proposed consolidation strategy with actual workloads from PlanetLab and compare it with other strategies based on different metrics.
The remaining sections of this article are as follows. Section 2 summarizes the current related work. Section 3 presents the system energy consumption model and the definition of each parameter. In Section 4, we introduce the proposed VM dynamic consolidation strategy. The experimental setup and the analysis of the experimental results are shown in Section 5. In Section 6, we summarize the article.

II. RELATED WORK
The dynamic consolidation of VMs can generally be divided into three parts: host load detection, the selection of VMs to be migrated, and the placement of VMs.
In the host load detection stage, the consolidation time needs to be determined based on the current load conditions. Beloglazov and Buyya [15] proposed five types of host overload detection: static threshold (THR), interquartile range (IQR), median absolute deviation (MAD), local regression (LR) and local robust regression (LRR) detection. The static threshold method uses a unified standard to handle overloaded hosts and cannot adapt to the current load changes. The other four algorithms can adaptively adjust the threshold according to historical load data to determine the migration timing for different load conditions. Fard et al. [22] used a temperature threshold to determine whether the host was overloaded. This algorithm monitors the temperature of all hosts and migrates VMs from overheated hosts to reduce the temperature. This algorithm can effectively control the performance and temperature of the host, but its disadvantage is the same as that of the traditional static threshold method; that is, it cannot effectively deal with load fluctuations. Yadav et al. [23] proposed an adaptive CPU utilization overload detection method based on Pearson's correlation coefficient. In the algorithm, the higher the level of resource utilization by VMs is, the higher the probability of host overloading. Prediction models play an important role in load detection. A predictive model can predict near-instantaneous changes in the load. By analyzing the current and future loads, we can determine the most reasonable migration time and reduce the number of invalid migrations. Li et al. [8] proposed a host overload detection method based on a robust linear regression prediction model, which enabled the host to complete the migration of VMs in advance before overloading. Compared with traditional linear regression models, several error correction methods are proposed for the model in this paper to improve the accuracy of the prediction results, and a safety factor is used to further reduce the SLAV rate. Hieu et al. [24] used linear functions for multiresource prediction at various times to estimate long-term utilization. Estimates of the long-term estimated resource utilization rate can be used to effectively perform overload and underload detection so that decision making can optimize system performance in a long period of time. Liu et al. [16] applied the autoregressive integrated moving average (ARIMA) model, which transforms an original time series into a stationary time series to predict the future host CPU utilization. Liu et al. [17] proposed an extreme learning machine for load prediction to establish a nonlinear model and predict changes in datacenter resources. Compared with linear programming prediction, this method can more accurately reflect variational trends.
In terms of VM selection, Beloglazov et al. [15] proposed four VM selection algorithms: minimum migration time (MMT), maximum correlation (MC), minimum utilization (MU), and random selection (RS) algorithms. During the process of VM migration, the service quality will be negatively affected. The MMT algorithm can reduce the time of VM migration and limit the performance degradation caused by migration. The MC algorithm considers the correlations among resources used by host applications. The higher the correlation is, the higher the probability of host overload will be. Therefore, the VM with the highest correlation with other VMs is selected for migration. Ammar et al. [7] wanted to maintain a balance among various host resources after VM selection, and they found that load-balanced hosts can accommodate more VMs than imbalanced hosts to prevent resource waste. Liu et al. [16] suggested than an overloaded host will cause SLAVs regardless of whether VM migration is performed. Therefore, a VM with a SLAV rate caused by migration that is less than the SLAV rate caused by overload should be selected for migration. This algorithm can find a balance between the two violations and reduce the overall violation level. Li et al. [25] considered the impact of VMs on host resources when selecting VMs. They selected the VM that contributed the most to the host load for migration, and the memory of the VM to be migrated was minimized.
The placement of VMs is similar to the multidimensional bin-packing problem. In [4], [8], [15], [24], [26]- [28], the authors proposed VM placement strategies based on a common bin-packing algorithm called BFD (best-fit decreasing algorithm). The BFD algorithm allocates VMs to the most suitable hosts in a specific order to minimize the consolidation time and complexity. In [4] and [15], the authors modified the BFD algorithm and proposed a power-aware BFD (PABFD) algorithm. PABFD selects the host with the smallest increase in energy consumption after allocation. Li et al. [17] proposed a VM placement strategy based on the resource capacity and energy consumption; specifically, the host with the largest resource capacity and lowest energy consumption is selected in each step as the target host. Hieu et al. [24] improved the PABFD algorithm proposed in [14] by adding a multitarget prediction module to reduce the possibility of host overload after VM migration. Li et al. [27] considered the impact of VM consolidation on VM switch traffic. The consolidation of VMs with excessively high sharing of virtual resources will limit the resources allocated by the VMs and reduce the performance of the VMs. Therefore, virtual switching overhead can be considered to effectively optimize the quality of service of VMs. Ammar et al. [7] considered balancing the resources of each computing node when integrating VMs. Song et al. [29] considered the resource demand during peak tasks based on the method of Ammar et al. [7]; the proposed system not only reserved certain resources to meet the peak resource demand but also balanced load resources. Mastroianni et al. [30] established allocation functions for the CPU and memory resources of hosts. When a host has considerable remaining resources, they are most likely to be selected as the target host for VMs. Xiao et al. [31] merged VMs based on alliance games. The algorithm divides hosts into ultrahigh-load hosts, high-load hosts and low-load hosts according to the utilization rate and redistributes them by splitting the VMs for ultrahigh-load hosts, combining VMs for low-load hosts, and combining VMs for low-load hosts and high-load hosts. This method can ensure that the utilization of each host is within a reasonable range during allocation, avoid the excessive utilization of a single host, and effectively limit SLAVs caused by the excessively aggressive traditional consolidation algorithm. Li et al. [25] used a constructed Bayesian network to predict the probability of a VM being migrated when the VM was deployed on a physical host with different load patterns and further estimated the total number of potential VM migrations on the physical host. When consolidating VMs, VMs were migrated to physical hosts with few potential migrations and sufficient available resources. This method considers the possible occurrence of SLAVs at future time points during consolidation, and it can maintain the stability of the system over a long period of time. Liu et al. [17], Feller et al. [32], and Xiao et al. [33] all adopted metaheuristic algorithms based on ant colonies for VM consolidation. A metaheuristic algorithm can effectively avoid the local optimal solution and optimize the overall result. Compared with the greedy algorithm, the performance of the metaheuristic algorithm is superior. Ferdaus et al. [13] used the ant colony algorithm to place VMs on the least-utilized host and improved utilization by balancing resources among hosts. This method can effectively reduce VOLUME 8, 2020 the energy consumption of datacenters but does not consider the quality of service, leading to SLAVs. Farahnakian et al. [14] proposed a dynamic consolidation algorithm for VMs based on ACS. The algorithm uses energy consumption, migration quantity and SLAVs metrics for multiobjective optimization. However, in the actual consolidation process, resulting in ineffective reduction of the number of migrations and an excessively long consolidation time for large datacenters.. Aryania et al. [34] made improvements to the approach of Farahnakian et al. [14]. They considered the number of sleeping hosts and the energy consumption of VMs during the migration process when performing VM consolidation, Additionally, the impact of memory on energy consumption during migration was accounted for. According to the experimental results, the algorithm in [34] performed better than the algorithm in [14] based on energy consumption, the SLAV and other metrics. Wu et al. [35] combined a greedy heuristic algorithm and a genetic algorithm and proposed an improvedgrouping genetic algorithm consolidation strategy to reduce energy consumption and migration costs. Gamal et al. [18] used a hybrid artificial bee colony and ant colony algorithm to optimize the dynamic environment of cloud computing. The experimental results verified that this strategy effectively balanced the load of each server, reduced energy consumption and the number of VM migrations, and improved service quality. Sharma and Reddy [36] used a hybrid genetic and particle swarm optimization algorithm to place VMs, and it reduced energy consumption and resource waste. The advantage of this method is that when the solution space is large and the adaptability level is low, the convergence speed of the genetic algorithm is slow. Notably, using the particle swarm algorithm to improve the adaptability of chromosomes can effectively reduce the time of VM consolidation and improve the consolidation quality. Liu et al. [37] considered the coordination problem for multiple VMs in a parallel application and proposed an initial virtual cluster allocation algorithm and a PM selection strategy based on PSO to reduce the overall transmission overhead, overall network resource consumption and total execution time for parallel applications. Wang et al. [38] imitated symbiotic survival and established a VM consolidation model. Biological competition at a same host will affect the host, and parasites will select a host with sufficient resources. This model can be well adapted to the consolidation of VMs. VMs at the same host are highly correlated and easily overloaded, and the performance of VMs can be ensured at hosts with sufficient resources. This article establishes a fitness function based on host utilization and the correlation of VMs and uses a biological heuristic algorithm to help VMs migrate to a suitable target host. Experiments have verified that this model can adapt to different datacenter loads and significantly reduce datacenter energy consumption and SLA violation rates. Ibrahim et al. [21] used the particle swarm algorithm to solve the problem of VM consolidation. The particle swarm algorithm can overcome continuity problems, so discretization is needed when using the algorithm to solve the box-packing problem. Discretization can lead to an insufficient number of particles in the population, and the algorithm may fall into a local optimal solution. Compared with [21], this paper adds a mutation subalgorithm to the algorithm, which mitigates the particle abundance issue. In addition, we introduce a local optimization algorithm to improve the particle movement direction.

III. ENERGY CONSUMPTION MODEL AND RELATED PARAMETERS A. ENERGY CONSUMPTION MODEL
The energy consumption model of the host is mainly determined by the energy consumption of the CPU, memory [10], disk and VM migrations [39], [40]. The energy consumption model is as follows: where E CPU , E mem , E disk and E migration are the energy consumption levels of the CPU, memory, disk and VM migrations, respectively. The energy consumption of a single host is mainly determined by the CPU, and the large-scale migration of VMs in a datacenter requires additional migration energy. Therefore, the energy consumption model in this paper only considers E CPU and E migration . The CPU energy consumption model can be represented as follows: (2) where P CPUi and P CPUm are the power consumption levels of the CPU for idle and full loads, respectively, and U CPU (t) is CPU utilization.
The energy consumption associated with VM migration mainly includes the energy consumption of the host and network bandwidth during migration [41], and the model can be expressed as: where P host is the maximum energy consumption of the source host and the target host during the VM migration process, P bw is the network bandwidth energy consumption, T mig i is the time required to migrate VM i, and N is the total number of VMs to migrate. Migration energy consumption is mainly influenced by the increase in host energy consumption during the migration process; therefore, in this article, we only consider P host . In addition, according to the model, it can be concluded that the migration time and the number of migrations have a considerable impact on the total energy consumption of migration and need to be considered during consolidation.

B. PARAMETERS
For a cloud datacenter, HL = {h 1 , h 2 , . . . , h n } represents a list of hosts at the datacenter, and VML i = {vm 1 , vm 2 , . . . , vm N } represents a list of VMs among hosts. Host resources (HR) and VM resources (VMR) can be defined considering the CPU, memory, network bandwidth and storage characteristics, where The CPU utilization of the host in this paper is defined as follows: where U vCPUi is the CPU utilization of VM i at the host, CPU vi is the MIPS allocated to VM i , and CPU h is the MIPS of the host.
The memory utilization of the host is defined as follows: where U vRAMi is the memory utilization of the VM i , RAM vi is the allocated memory capacity of the VM i , and RAM h is the memory capacity of the host. The network bandwidth utilization of the host is defined as follows: where BW used is the network bandwidth used by the host and BW h is the total network bandwidth of the host. The storage utilization of the host is defined as follows: where STORAGE used is the used storage capacity of the host and STORAGE h is the total storage capacity of the host.

C. MIGRATION COST
During the process of the live migration of VMs, although the VMs are not suspended, live migration has a negative impact on performance. According to the research in [10], the average performance degradation associated with VMs during the migration process is approximately 10% of the CPU utilization rate. Each migration may cause SLAVs and affect the operating performance of the system. Therefore, it is important to reduce the migration time and minimize the number of VM migrations. The VM migration cost model is as follows: where T mig i is the time required for the migration of VM i , RAM vi is the memory used by VM i , BW h is the available network bandwidth of the host, U deg i is the overall performance degradation associated with VM i , and u i (t) is the CPU utilization of VM i .

IV. DYNAMIC MIGRATION STRATEGY OF VMS BASED ON A GRAY MODEL AND IMPROVED DISCRETE PARTICLE SWARM OPTIMIZATION
The proposed strategy can be divided into three parts: host load detection, the selection of VMs to be migrated, and VM placement. During host load detection, we set high and low thresholds with a gray prediction algorithm and dynamic threshold adjustment strategy to identify overloaded and underloaded hosts. In the process of VM selection, VMs are selected according to the proposed VM selection algorithm based on the MMT and CPU utilization. In the process of VM placement, considering load balancing among datacenter components, we adopt a VM placement strategy based on improved discrete particle swarm optimization to balance each host and improve system performance.

A. HOST LOAD DETECTION
The energy consumption and performance of a cloud datacenter are closely related to the load conditions. If the load is too high, insufficient resource space will be available on the host, which may cause SLAVs and affect the performance of VMs. If the load is too low, the resources on the host may not be effectively used, resulting in excess energy consumption. In this paper, we set the high threshold (T high ) and low threshold (T low ) of CPU utilization as the migration thresholds. If the CPU utilization level is higher than the high threshold, the host is overloaded, and some VMs must be migrated; if the CPU utilization level is lower than the low threshold, the host is underloaded, and all VMs need to be migrated. After migration, a sleep state can be used to reduce the number of hosts running at the datacenter. During the host overload detection process, due to the dynamic fluctuations in the load, a host may be overloaded, but the load will return to normal relatively quickly. This issue may cause excess migrations or invalid migrations, resulting in increases in energy consumption and migration costs. To solve this problem, we propose a gray prediction algorithm with residual correction to predict the loads of hosts in the future [19], [42]. The gray prediction algorithm is an exponential prediction model that uses a small amount of incomplete information to establish a mathematical model and make predictions. The algorithm predicts future CPU utilization based on historical load data. The process of the gray prediction algorithm is as follows: First, we define the sequence: where x (0) represents the historical CPU utilization data for the host. GM (1,1) model: The gray system is a differential equation established for discrete sequences, and GM (1,1) is a first-order differential VOLUME 8, 2020 equation model. The form of GM (1,1) is as follows: Discrete forms of the differential equations are established: x (1) In general, the value of µ is 0.5: Next, (13) is transformed into a matrix expression: This equation can be rewritten as The prediction formula can then be obtained based on the above formula: To improve the accuracy of prediction, we use residuals to establish the GM (1,1) model and modify the prediction results. A residual is defined as follows: The GM (1,1) model for residuals iŝ whereŷ (0) (k) is the predicted value of CPU utilization at the next time step. Li et al. [8] and Liu et al. [16] proposed a strategy to migrate VMs before a host is overloaded and reduce SLAVs, but we take another approach. When both the host CPU utilization and predicted utilization levels are higher than the high threshold, the VMs at the host need to migrate; otherwise, migration will not occur. This method can reduce the impact of load fluctuations on the migration accuracy.
To further reduce the performance degradation caused by VM migration, we use a dynamic threshold adjustment mechanism to optimize the consolidation timing. After a host is determined to be overloaded based on the high threshold and gray prediction, if the load fluctuation is small for the host, there is a small probability of an SLA violation occurring. In this case, the high threshold will be increased, and the migration threshold of the host will be changed to T low , T high + T . At this time, the system can reduce the number of VM migrations when the load is relatively stable. To ensure the performance of the host and the VMs, the adjusted high threshold should not be higher than 0.9. We use the MAD to indicate the load fluctuations for the host over a period of time. By assuming U i = {U i (1), U i (2), . . . , U i (n)} is the historical CPU utilization set, the MAD can be defined as follows: Set the load fluctuation limit as S. If MAD(i) is less than S, increase the high threshold by T ; otherwise, the threshold will not change.

Algorithm 1 Host Overload Detection
Input: HostList Output: OverUtilizedHostList 1 OverUtilizedHostList ← NULL; 2 for each host in HostList do 3 if MAD(UtilizationHistory(host)) < S then 4 T ← T high + T ; 5 else T ← T high ; 6 end if 7 if utilization(host) > T && Gray(utilization 8 (host)) > T then 9 OverUtilizedHostList.add(host); 10 end if 11 end for 12 return OverUtilizedHostList Algorithm 1 and Algorithm 2 describe the specific process of load detection. In Algorithm 1, a reasonable high threshold is determined based on the historical fluctuations in the load (lines 2-6). We use a gray prediction algorithm to determine whether a host is overloaded. If both the current load rate and the predicted load rate are higher than the high threshold, the host is determined to be overloaded (lines 7-10). In Algorithm 2, a list of hosts to be detected is created and all overloaded hosts, sleeping hosts, and hosts in migration are removed from the list (lines 2-5). When selecting an underloaded host, the host with the lowest utilization rate will be selected at each time step (lines 7-13).

B. VM SELECTION
For overloaded hosts, some VMs must be migrated until the CPU utilization level of the host is lower than the high threshold. A previous paper [15] proposed an MMT algorithm to migrate VMs. This algorithm selects the VM with the smallest T m i to migrate each time. We consider CPU utilization based on the MMT algorithm; thus, at each step, VM migration reduces the CPU utilization of the host as much as possible, and the migration time can be reduced. According to the proposed algorithm, VMs with low migration times and high MIPS values will be the most likely to be selected. V (h i ) is set for all the VMs on host i , and the proposed selection algorithm is defined as follows: where  (28), and the VM with the smallest calculated index is selected for migration (lines 6-12). After each step in the VM selection process, it must be determined whether host overloading is still occurring after migrating VMs. If the host is overloaded, the VM selection process continues; otherwise, VM selection ends (lines 13-17).

C. VM PLACEMENT
For overloaded and underloaded hosts, all VMs to be migrated should be allocated to new hosts. First, the selected VMs from the overloaded hosts are migrated, and then the VMs from the underloaded hosts are migrated. Wang and Tianfield [6] and Liu et al. [17] considered only the CPU utilization when placing VMs, and the performance of VMs was assessed based on multiple parameters. Therefore, considering multiple resources when integrating VMs can improve the rationality of the consolidation strategy. We consider CPU, memory, network bandwidth and storage resources, so the placement of VMs can be regarded as a multidimensional VOLUME 8, 2020 bin-packing problem. If the VMs at a datacenter can be evenly distributed among hosts, the probability of SLAVs occurring at a host will be reduced. To ensure that the VMs can balance the load of the datacenter and rationally use resources after the consolidation of VMs, this paper proposes a balanced placement strategy for VMs based on the discrete particle swarm algorithm.
The strategy we propose uses the absolute error of the mean to assess the load balance for the datacenter: where U R represents the utilization of a certain resource by each host and MEAD(U R ) represents the mean absolute deviation of U R . The objective function of the VM placement strategy is as follows: The goal of the placement strategy is to minimize the value of the objective function. According to the objective function, we propose an improved discrete particle swarm optimization (DPSO) method.
Particle swarm optimization is a biology-based metaheuristic evolutionary computational method. The algorithm is derived from studies of bird predation behavior, and the basic objective is to find the optimal solution through collaboration and information sharing among individuals in a group [21]. Particle swarm optimization simulates the birds in a swarm as massless particles. These particles have only two properties: speed and position. Each particle separately searches for the optimal solution in the search space and records it as the current individual optimal solution. The individual optimal solution is shared with other particles in the particle swarm to find the best individual optimal solution as the current global optimal solution. Each particle in the swarm adjusts its speed and position according to the current individual optimal solution and the current global optimal solution [43].
When DPSO is used to solve the global optimization problem, multiple feasible solutions for VM placement are combined, and each feasible solution is a particle. Each particle in the population has a position vector and a velocity vector [44]. The variables associated with each dimension of a particle are continuously updated in an iterative process until the global optimal position is determined. The position and velocity of each particle are defined as follows: where X i is the position vector of particle i, x i,1 is the ordinal of the host where the first VM for particle i is placed, V i is the velocity vector of particle i, and the elements in the vector are the particle speeds for all dimensions.
In the population initialization phase, N feasible VM placement schemes should be randomly generated. To guide the particles toward the globally optimal position, this paper proposes a resource-aware VM placement strategy. The scheme obtained by this strategy is used as one of the particles in the initial population.
The resource-aware VM placement strategy is based on the remaining available resources for each host. The definitions of the resource and host selection strategies are as follows: In the particle search process, the optimal position obtained by particle i is called the individual optimal solution of particle i, denoted as Pbest i = [p 1,i , p 2,i , . . . , p n,i ]; the global optimal position found by all particles in the population is called the global optimal solution, denoted as Gbest = [g 1 , g 2 , . . . , g n ]. The fitness value for each particle is calculated according to the objective function, and the individual optimal solution and the global optimal solution are updated at each iteration. The particles continuously update their speed and position according to the individual optimal solution and the global optimal solution, and the updating method is as follows [45]: where v t i,d is the velocity of particle i in the d-th dimension at the t-th iteration, x t i,d is the position of particle i in the d-th dimension at the t-th iteration, p t i,d is the individual optimal solution of particle i in the d-th dimension at the t-th iteration, g t d is the global optimal solution in the d-th dimension at the t-th iteration, ω is the inertia factor, c 1 and c 2 are learning factors, rand() is a random number between 0 and 1, v max is the maximum velocity of particles, which is taken as 20% of the number of hosts in this paper, m 1 and m n are the minimum ordinal and maximum ordinal of the host, respectively, and round(v t+1 i,d ) represents the rounding of v t+1 i,d . Because the host ordinal is an integer, the continuous quantity needs to be converted into a discrete quantity, so the range of x i is an integer between m 1 and m n . If the updated result is greater than m n , x i is m n ; if the updated result is less than m 0 , x i is m 0 . To improve the global search ability of the particles in the early part of the search process, avoid falling into local optima, increase the local search ability in the late part of the search process, and accelerate the convergence speed, the inertia factor and the learning factor are dynamically adjusted in a linear decreasing manner: where ω ini and ω end are the maximum and minimum values of ω, respectively; c 1,ini and c 1,end are the maximum and minimum values of c 1 , respectively; c 2,ini and c 2,end are the maximum and minimum values of c 2 , respectively; t is the current number of iterations; and T is the maximum number of iterations.
To increase the diversity of particles, we propose a particle mutation strategy. If the fitness value of particle i is less than the average fitness value of the particle swarm, map the particle speed to the interval [0,1]; then, mutate the particles with a certain probability. The specific implementation is as follows: s(v i,d ) represents mapping v i,d to the interval [0,1]; M is the mutation probability of the particles; M ini and M end are the maximum mutation probability and the minimum mutation probability, respectively; and random(m 0 , m n ) is a random number between the host numbers. Particle mutation is more likely to occur in the early part of the iteration process than in the later part, which is helpful for expanding the solution space of particles and avoiding local convergence; as the number of iterations increases, the probability of mutation will gradually decrease, making the particles converge quickly.
Algorithm 4 describes the specific process of the particle swarm consolidation algorithm. First, initialize the particle swarm parameters and the VM placement plan, as suggested in (34) (lines 1-2). By calculating the fitness function, determine the individual optimal position and the global optimal position of the initial particle swarm (lines 3-9). In the algorithm process, the velocity and position of each particle at each iteration are updated through (35) and (36), a new VM placement plan is generated, and the corresponding fitness function value is calculated (lines [11][12][13][14][15][16][17][18][19][20][21]. For a plan with a fitness value lower than the average value, the placement

Algorithm 4 MDPSO VM Placement
Input: HostList, vmsToMigrate Output: MigrationMap 1 Initialize algorithm parameters 2 Initialize the population based on HostList and vmsToMigrate, and use eq. (34) as one of the initial particles 3 Calculate F based on eq. (30) plan needs to be updated through a mutation operation to ensure the diversity of the placement plan (lines 23-28). Finally, according to the results of the iteration, the individual optimal position and the local optimal position are calculated, and the optimal placement plan for the VM is obtained (lines 29-43).

V. PERFORMANCE EVALUATION A. EXPERIMENTAL SETTING
This article uses CloudSim as the simulation platform to test the performance of the proposed strategy. We simulated a datacenter consisting of 800 heterogeneous physical hosts in the experiment, including 400 HP ProLiant ML110 G4 servers (Intel Xeon 3040, 2 cores * 1.86 GHz) and 400 HP Pro-Liant ML110 G5 servers (Intel Xeon 3075, 2 cores * 2.26 GHz). There were four types of VMs in the experiment: high-CPU medium-instance (2500 MIPS, 0.85 GB), extralarge-instance (2000 MIPS, 3.75 GB), small-instance (1000 MIPS, 1.7 GB), and micro-instance (500 MIPS, 613 MB) VMs. To test the performance of the algorithm in an actual environment, the workload data used in the simulations were provided by the PlanetLab project [15]. PlanetLab is a computer cluster project distributed around the world. The project collects CPU utilization data for VMs from servers in more than 500 locations around the world. The project measures utilization every five minutes, and each measurement period is one day. We selected the workload traces for 10 days from March to April 2011. The specific data are shown in Table 1.
The parameters of the DPSO method used in this paper are shown in Table 2.

B. PERFORMANCE METRICS 1) ENERGY CONSUMPTION
Reducing the energy consumption of cloud datacenters is one of the goals of the VM dynamic consolidation algorithm proposed in this paper. According to the energy consumption modeling in Chapter III, the energy consumption of the host is closely related to CPU utilization. Table 2 shows the relationship between energy consumption by the test servers and CPU utilization.

2) NUMBER OF VM MIGRATIONS
VM migration requires a certain amount of network bandwidth resources and has a negative impact on the performance of VMs during the migration process. Excessive migrations can lead to an increase in SLAVs. Therefore, the number of VM migrations should be reduced to ensure the service quality of the datacenter.

3) SLAV TIME PER ACTIVE HOST (SLATAH)
SLATAH represents the percentage of time that active hosts experience a CPU utilization level of 100% [15]. In this case, the application performance of each VM is limited by the capacity of the host, and the VM cannot meet the required performance level. The definition of SLATAH is as follows: where N is the number of hosts; T SLAV i is the total time that host i has experienced a utilization level of 100%, resulting in SLAVs; and T active i is the total active time for host i .

4) PERFORMANCE DEGRADATION DUE TO MIGRATION (PDM)
PDM is the overall performance of degradation for VMs due to migration [15]. The definition of PDM is as follows: where M is the number of VMs; C deg i is the estimated value of the performance degradation of VM i due to migrations, accounting for approximately 10% of the CPU utilization; and C cap i is the total CPU capacity requested by the VM i during its active time.

5) SLAVs
SLAVs for users occur due to VM consolidation. SLAVs are related to the SLATAH and PDM, and they reflect a service quality degradation caused by host overload and VM migration. The definition of a SLAV is as follows: ESV is the product of energy consumption and SLAVs, and it is a comprehensive performance metric for the VM consolidation strategy. The definition of ESV is as follows:  [14] and AVVMC [13], as well as the DVFS strategy. Table 4 shows the average of all metrics obtained after using 10 sets of data for all strategies.
According to the data in Table 4, VM consolidation is vastly superior to DVFS based on energy savings, and the proposed GM-DPSO strategy is better than other dynamic consolidation strategies for all metrics. In terms of energy consumption, compared with other dynamic consolidation strategies, GM-DPSO can reduce energy consumption by up to 34.53%. In terms of service quality, the GM-DPSO algorithm reduces SLATAH and PDM by up to 68.62% and 90.29%, and the overall SLAV and ESV values are reduced by 97.53% and 98.26%, respectively. ACS-VM is superior to AVVMC in terms of performance. Compared with ACS-VM, GM-DPSO can reduce energy consumption, the VM migration number, SLAVs and ESV by 7.61%, 89.9%, 94.17% and 94.74%, respectively. According to the average experimental results, GM-DPSO achieves the best performance in terms of energy consumption and service quality, and provides the most optimal service quality. To specifically analyze the performance of the dynamic consolidation algorithm under different load conditions, we select ''20110306'' data (number of VMs: 898; average utilization: 11.44%) as a low-load case and select ''20110403'' data (number of VMs: 1463; average utilization: 12.93%) as a high-load case. Figure 1 shows the performance of various dynamic VM migration strategies in terms of energy consumption. Under low and high loads, GM-DPSO can reduce energy consumption by up to 34.43% and 32.36%, respectively. In addition to GM-DPSO, LR-MC and LRR-MC obtain the best performance in terms of energy consumption (20.03% higher than GM-DPSO under low loads; 17.64% higher than GM-DPSO under high loads). The Iqr-Mmt-1.5 strategy has the highest energy consumption, and the methods that use the MMT strategy in the VM selection algorithm are inferior to those that use the MC algorithm based on energy consumption because the MMT algorithm only considers the migration time when selecting VMs and does not consider the impact of the selected VMs on reducing energy consumption. MC considers the relations among VMs and can quickly reduce host CPU utilization. Compared with the ACS-VM and AVVMC algorithms, GM-DPSO can reduce energy consumption by 8.68% and 14.28%, respectively, under low-load conditions. Under high-load conditions, the three algorithms perform similarly. Both ACS-VM and AVVMC place VMs on as few hosts as possible to reduce energy consumption. The GM-DPSO strategy proposed in this paper greatly reduces the number of VMs to be migrated (as shown in Figure 2) and the energy consumption of VMs. Dynamic threshold adjustment helps hosts with stable loads to accommodate additional VMs. The GM-DPSO algorithm effectively reduces the number of running hosts and reasonably allocates resources among hosts, thereby reducing the number of times a host is switched. Therefore, the algorithm has certain advantages in reducing energy consumption.
As shown in Figure 2, GM-DPSO can greatly reduce the number of VM migrations. According to the test results, the average number of migrations for all benchmark algorithms is 26567.05, and the average number of migrations for GM-DPSO is only 2303. GM-DPSO introduces gray prediction and dynamic threshold adjustment for overload detection, thus effectively avoiding invalid migrations. The placement strategy based on load balancing reduces the possibility of a host being overloaded and decreases the number of subsequent migrations. In addition, this article reduces the lower threshold in low-load detection to prevent excessive VM consolidation. Figures 3, 4 and 5 show the performance of different consolidation algorithms in terms of SLAs. These figures show that GM-DPSO can effectively avoid excessive consolidation by considering the balance of the load distribution, thus reducing the probability of resource shortages for some nodes. GM-DPSO can reduce the overall SLAV value by up to 97.67% and 98.48% under low and high loads, respectively; SLATAH caused by host overload can be reduced by up to 68.77% and 75.58%; and PDM caused by VM migrations can be reduced by up to 90.91% and 90%. Compared with other traditional algorithms, ACS-VM and AVVMC are superior based on SLAV, but the performance of GM-DPSO is  generally superior. In addition, the two ant colony consolidation algorithms focus on reducing energy consumption at the datacenter, and the consolidation strategy adopted does not significantly reduce the number of VMs to be migrated;  thus, these methods are not as good as GM-DPSO in terms of SLAV.
SLATAH reflects the degradation of service quality due to host overload. The LR and LRR algorithms have high SLATAH metrics under different loads, but they perform well in terms of energy consumption. The LR and LRR algorithms encourage the consolidation of VMs, but they also increase the possibility of host overload. The MMT algorithm is superior to the MC algorithm based on SLATAH because the MMT algorithm reduces the time for VM migrations and excessive CPU use during the migration process, thus decreasing the host overload rate. We propose a VM placement strategy based on improved DPSO to consolidate VMs. This approach fully considers load balancing when placing VMs, thus ensuring that the remaining resource space for each host is balanced and preventing the overload of hosts caused by the centralized placement of VMs. In addition, the lower threshold limits the number of VMs to be migrated, which reduces the high load rate of a host due to the excessive consolidation of VMs.
The PDM is closely related to the number of VM migrations. According to the analysis results, gray prediction and dynamic threshold adjustment can reduce the migration of VMs to decrease the PDM. In addition, the VM placement strategy of GM-DPSO not only reduces the host overload rate but also decreases VM migration caused by overload and improves PDM performance. The VM selection strategy proposed in this paper considers the migration time and CPU utilization, so it can synchronously optimize SLATAH and PDM. Based on the above analysis, SLATAH and PDM are closely interlinked. A SLATAH value that is too high increases the number of overloaded hosts and further increases the number of VMs to be migrated. The migration of underloaded hosts can reduce the number of servers operating at a datacenter, and the migration of VMs increases the load rate of active hosts and the probability of host overload.
ESV reflects the overall levels of energy consumption and service quality, as shown in Figure 6. Under different load conditions, GM-DPSO is superior to all other strategies in terms of the ESV result. Thus, GM-DPSO can effectively guarantee the service quality of a datacenter while reducing energy consumption. The proposed strategy provides high-performance cloud computing and balances the remaining resource space for each computing node, thereby improving the scalability of the system and reducing the operating cost. Therefore, a win-win situation is achieved for users and suppliers.

VI. CONCLUSION
In this article, we propose a dynamic VM consolidation (GM-DPSO) strategy based on a gray model and an improved discrete particle swarm algorithm. First, we use the gray prediction algorithm and dynamic threshold adjustment mechanism to determine the timing of migration. Then, we select the VMs to be migrated according to the migration times and CPU utilization levels and complete the consolidation with a VM placement strategy based on the improved discrete particle swarm algorithm. We analyzed the factors that affect the quality of system service and developed a corresponding combination strategy. We use actual load tests to assess the performance of the consolidation strategy and compare GM-DPSO with multiple algorithms. The experimental results show that the GM-DPSO algorithm can effectively reduce the energy consumption of the system and improve the quality of service under different load conditions. The overload detection strategy based on gray prediction and dynamic threshold adjustment can effectively reduce the migration of VMs and SLAVs. The DPSO VM placement strategy searches for the optimal solution in the global scope. During the search process, the algorithm dynamically adjusts various parameters and introduces a mutation factor to increase the richness of the population and avoid local optima. The DPSO strategy provides a reasonable allocation method for datacenter resources, reduces the probability of service quality degradation, and improves the service stability of the system. Overall, the combined algorithm adopted by GM-DPSO greatly reduces the energy consumption of VM migration and effectively decreases the number of hosts running at a datacenter, thereby reducing the energy consumption of the entire system.
At an actual datacenter, to ensure the stability and security of the operation, backups are required for each host. The host backup can transfer data to the backup host when the host experiences sudden downtime to continue to serve users. Therefore, in future work, we hope to consider sufficient backup redundancy in the consolidation algorithm so that it can meet the needs of practical work and be used at actual datacenters.
YIFAN SHAO received the B.S. degree from the School of Electrical Engineering, Nanjing Institute of Engineering, in 2018. He is currently pursuing the master's degree with the School of Electronics and Information Engineering, Shanghai University of Electric Power. His research interests include cloud computing and virtual machine (VM) consolidation.
QIANGQIANG YANG received the M.S. degree from Shanghai Jiao Tong University. He is currently pursuing the Ph.D. degree in communication and information engineering with Shanghai University. He is also the Director of the School of Electronic Information Engineering, Shanghai University of Electric Power. His current research interests include cloud computing, big data, and artificial intelligence.
YAJUAN GU received the master's degree from Shanghai Jiaotong University. She currently works as the Deputy Director of CCSSR. Her current research interests include energy saving, emission reduction, and green environmental protection. VOLUME 8, 2020 YONG PAN received the B.S. degree from Tiangong University. He currently works as the IT Manager at CCSSR. His current research interests include cloud computing, big data, and artificial intelligence.
YI ZHOU received the B.S. degree from Jinan University. He currently works as the General Manager with MXSun. His current research interests include cloud computing, big data, and artificial intelligence.
ZIAO ZHOU is currently pursuing the bachelor's degree with the School of Electronical Information Science and Technology, Kunming University of Science and Technology. His current research interests include microcomputer principles and computer network principles.