Real-Time Workflows Oriented Hybrid Scheduling Approach With Balancing Host Weighted Square Frequencies in Clouds

High energy consumption in cloud data centers has become one of the main obstacles to green cities, and an urgent problem to be solved. So far, a large number of scheduling algorithms have been developed to reduce energy consumption for executing workflows. However, most existing algorithms have obvious defects in energy and resource efficiency, because they schedule workflow tasks to hosts directly and ignore that the host is so powerful that a single workflow task cannot make full use of its resources. To resolve the issue, a new scheduling architecture is designed for cloud data centers. Then, two principles are derived: one suggests that hybrid scheduling tasks from different workflows is helpful for improving resource utilization; the other one deduces that balancing host weighted square frequencies can minimize total power consumption of active hosts under a given resource requirement. On the basis of the scheduling architecture and the two principles, an oNline schEduling AlgoriThm, namely NEAT, is proposed to schedule dynamic workflows with deadlines. Furthermore, three strategies for dynamically adjusting the available virtual machines (VMs) and active hosts are proposed and integrated into NEAT to improve the energy and resource efficiency for cloud data centers. Finally, the proposed NEAT is compared with three existing algorithms using real-world workflow traces to demonstrate its superior performance with respect to energy and resource efficiency while guaranteeing the timing requirements of workflows. Compared with baseline algorithms, NEAT is capable of reducing energy consumption for cloud data centers by an average of 40% in the range from 34.0% to 45.67%.


I. INTRODUCTION
With the soaring expansion of cloud computing services, the number of hosts for supporting these services in cloud data centers is increasing sharply [1]. The number of hosts in a cloud data center is up to tens of thousands, or even hundreds of thousands. 1 It is inevitable that such large size of hosts expend massive electric energy, which comprises around 1.5% of the global energy consumption [2], [3]. Nowadays, the increasing energy costs [4] and negative environmental impacts have pressured cloud providers to improve energy efficiency for their cloud data centers.
The associate editor coordinating the review of this manuscript and approving it for publication was Amjad Gawanmeh . 1 https://github.com/google/cluster-data Contrasting to high energy consumption, the resource utilization in cloud data centers is very low, and the average value is around 15% [5], [6]. What is worse, hosts in low resource utilization still have very high energy powers, even some idle hosts dissipate more than 60% of their peak energy powers [6], [7]. Thus, a considerable proportion of energy in cloud data centers can be saved by improving hosts' resource utilization and shutting down idle hosts.
The reason for such low resource utilization of hosts is threefold. Firstly, it is unusual for a single workflow task to fully use the resources of a powerful host, whose average resource utilization is less than 40% when executing a workflow task [8]. Then, the workflow applications in cloud data centers typically comprise of a great deal of tasks with data dependencies [8]- [10], e.g., Montage and CyberShake.
The data dependencies among tasks inevitably result in a large number of idle time slots on resources, which further reduce resource utilization. Thirdly, workflow applications are submitted by a wide range of end-users dynamically, the data centers' workloads will fluctuate over the time. For many cloud data centers, their peak workloads are often many times their average workloads [6], [7], thus their resources' over-provisioning is inevitable at non-peak time.
Motivation: Most existing works often neglect that a single workflow task cannot fully use a host's resource, and directly allocate workflow tasks to hosts for execution, causing significant low resource efficiency. What is worse, the existing works rarely take full advantage of the idle time slots on resources. In this paper, we strive to incorporate the advantages of virtualization technology and the elasticity of clouds into energy-efficient scheduling approach. Specifically, with the virtualization technology, a host is split into several independent units (i.e., VMs) to execute multiple workflow tasks simultaneously, and the available VMs can be scaled up/down dynamically to adjust the number of active hosts. Besides, the fact that tasks from different workflows have not data dependencies is explored to further improve resource utilization.
Contributions: The main technical contributions are as follows.
• A novel reactive scheduling architecture is designed to transform an online scheduling problem to multiple static scheduling processes. This scheduling architecture also enables each static scheduling process to reschedule all the waiting workflow tasks by taking full advantage of all available information at each scheduling time.
• We derive two principles: 1) executing tasks from different workflows alternately at the same VMs is helpful for improving VMs' resource utilization; 2) the total power consumption is minimized by balancing weighted square frequencies of hosts, when the resource requirements and active hosts are given.
• Based on the scheduling architecture and the two principles, we propose an energy-efficient oNline schEduling AlgoriThm, NEAT, to improve the energy and resource efficiency for executing dynamic workflows with deadlines.
• We conduct three groups of experiments using realworld workflow traces to compare the proposed NEAT with other three existing algorithms. The paper is organized as follows. Section II reviews the relevant literatures. Section III introduces the scheduling architecture and problem formulation, followed by detailing the proposed algorithm in Section IV. Section V reports the experimental results. Section VI concludes this paper.

II. RELATED WORK
Reducing energy consumption for executing workflows in clouds has become one of the hottest directions. In past years, many energy-aware workflow scheduling algorithms have been reported, and these existing algorithms can be roughly classified as 1) population-based evolutionary algorithms and 2) heuristic algorithms.
The population-based evolutionary algorithms have become an attractive optimization technique for resolving the workflow scheduling problems. For instance, Mezmaz et al. employed genetic algorithm to make a trade-off between finish time and energy consumption for workflow scheduling in clouds [11]. Yao et al. employed a multi-swarm algorithm to simultaneously optimize makespan, cost, and energy consumption for workflows in clouds [12]. Zhang et al. proposed a bi-objective genetic algorithm to schedule workflows in clouds, to minimize energy consumption and enhance system reliability [13]. However, these population-based evolutionary approaches have much higher time complexity, and are not suitable for scheduling dynamic workflows in clouds.
With respect to the heuristic algorithms, they can be further divided into three categories: list-based, clustering-based, and duplication-based approaches. For list-based workflow scheduling algorithms, they mainly include two stages: assigning weights to tasks and schedule tasks in descending order of their weights. For example, Topcuoglu et al. extended the classical list-based workflow scheduling algorithm HEFT [14] to search for Pareto solutions with respect to finish time and energy consumption [15]. Tao et al. designed a parallel intelligent algorithm to maximize the reliability, minimize the execution time, cost and energy [16]. Schranzhofer et al. developed an algorithm to map tasks to processing resources, striving to minimize the expected average power consumption [17]. Regarding clustering-based scheduling algorithms, they firstly group all the tasks in a workflow into many clusters, and then schedule all tasks in each cluster [18], [19]. The fact that there exists no data transmission among tasks on the same computing resources is exploited by duplication-based algorithms to improve resource efficiency and workflows' finish time. For example, Zong et al. developed two tasks duplication based scheduling strategies to reduce makespan and energy consumption for executing workflows on clusters [20]. However, these existing works focused on scheduling a single workflow, and directly scheduling workflow tasks to hosts for execution. In this work, tasks from different workflows having no data dependencies is explored to schedule tasks from many workflows in a hybrid way. Besides, the varied resource requirements of workflow tasks and the changing workload of cloud data centers are also considered in this work.
So far, we have published several papers on workflow scheduling in clouds [21], [22]. Compared with the prior works, this paper derives two principles to improve resource utilization, minimize total power consumption. Then the scheduling algorithm is developed with three novel scale down strategies. Moreover, the time complexity of the proposed algorithm is detailed.

III. MODELING AND PROBLEM FORMULATION
In this section, the models of workflows, scheduling architecture and workflow scheudling are introduced.

A. WORKFLOW MODELING
The dynamic workflows in a cloud data center can be characterized by W = {w 1 , w 2 , . . . , w m }, and w i (i = 1, 2, · · · , m) stands for the i-th workflow. The workflow w i can be formulated as where a i , d i and G i denote w i 's arrival time, deadline and structure, respectively. For workflow w i , its structure G i can be further described by a directed acyclic graph (DAG) . . , t i |T i | } represents the task set in workflow w i ; E i ⊆ T i × T i indicates the data dependencies among workflow tasks. An edge e i pj ∈ E i means the start of task t i j depends on the output data of executing task t i p , then t i j and t i p call the other side as immediate successor and predecessor. All the immediate successors of task t i j is modeled as pred(t i j ), and all its successors are denoted as succ(t i j ).

B. SYSTEM MODELING
One cloud data center typically comprises of large-scale heterogeneous hosts, i.e., H = {h 1 , h 2 , . . . , h n }, where n denotes the host count. For a host h k ∈ H , it can be modeled as where m k , sc k , n k , and p max k denote h k ' memory size, storage capacity, network bandwidth, and peak energy power, respectively; the discrete set of voltages-frequency pairs of host h k is described as , v max k )}; the VMs on host h k is modeled as VM k = {vm k,1 , vm k,2 , · · · , vm k,|VM j | }, and vm k,l stands for the l-th VM on host h k . For VM vm k,l , it can be further depicted as vm k,l = {f k,l , m k,l , s k,l }, where f k,l , m k,l and s k,l represent the requirements of vm k,l 's CPU frequency, memory and storage capacity, respectively. With virtualization technology, a host's hardware resources can be separately shared by many VMs, and these VMs can migrate among different physical hosts.
In this work, we propose a new reactive scheduling architecture for deploying workflows on clouds, which is shown in Fig. 1. Similar to traditional multiprocessor systems, the scheduling architecture in a cloud data center can be divided into three layers: applications, scheduling and resources. Throughout the architecture, the scheduling layer bridges the application layer and resource layer. However, differing from traditional systems, the resource layer in clouds consists of two layer: VM layer and host layer. Moreover, the available VMs in VM layer can be scaled up/down dynamically based on the workload of data centers.
The main focus of this work is the scheduling layer, which consists of the following four components: 1) a task pool (TP) which contains all the waiting workflow tasks; 2) a schedulability analyzer whose function is to produce the mappings between waiting workflow tasks and VMs, and the plan of adjustment for the hosts and VMs; 3) a task controller that implements the plan of schedulability analyzer, and allocates tasks to corresponding VMs; 4) a resource controller which executes the adjustment plan of resources. The novelty of this scheduling architecture mainly embodies that all the waiting workflow tasks are queuing in the TP rather than on the VMs directly, and only tasks that are ready to be executed will be allocated to VMs. This architecture makes it easy for the clouds to obtain all waiting tasks upon the arrival of new workflows, and reschedule those tasks, together with the tasks from the new workflows, for optimal scheduling solutions. With this architecture, the online scheduling problem is simplified into a sequence of static scheduling problems.

C. ENERGY MODEL OF HOSTS
Assuming that p max k and s k are the maximal energy power and the ratio of static energy power for host h k . Host h k 's energy power can be described as [7]: where y t k ∈ {1, 0} means whether h k is active at time t; f max k and f k are h k 's maximum CPU frequency and working CPU frequency at time t, respectively. Then, the energy consumption (EC) of n hosts from time instant st to et is computed as follows.
where parameters y t k and f k are time-dependent variables, and other parameters are fixed values for host h k .

D. PROBLEM FORMULATION
We use the record variable x i j,kl to indicate the mapping relationship between task t i j and VM vm k,l on host h k . The parameter x i j,kl is 1 if task t i j is mapped to VM vm k,l , otherwise, x i j,kl equals to 0, i.e., The data dependencies among tasks in each workflow comes to the following constraint: where ft i p,kl denotes task t i p 's finish time, and tt i pj stands for the data transmission time between task t p and task t i .
Once all the tasks in w i are scheduled to VMs, w i 's finish time ft i refers to the maximum finish time of all these tasks.
Another constraint in workflow scheduling comes from meeting workflows' deadlines, which is formulated as follows.
In addition, the limited resource of a host also restricts the number of VMs on it, which causes the following constraint.
Satisfying the constraints in formulas (4), (6) and (7), the first optimization objective is to minimize the energy consumption for completing the workflow set W , i.e., where n denotes the host count; st and et are the start time and the end time for executing the workflow set W , respectively.
Apart from the energy consumption, resource utilization is another important metric to be optimized. Thus, we try to maximize hosts' average resource utilization: where m and |T i | represent the workflow count of W and the task count of workflow w i , respectively; cpu i j and ct i j are task t i j 's CPU frequency requirement and computation time; n denotes the number of hosts in clouds; and wt k represents host h k 's active time.

IV. ALGORITHM DESIGN
Since workflow scheduling on multiple machines belongs to typical NP-complete problem, its optimal solutions can be gained only for the problems with small size. Therefore, we propose a reactive heuristic method to make schedulability analysis for dynamic workflows with deadlines, followed by the algorithm description and its time complexity.

A. PRELIMINARIES
Before scheduling workflows, how to rank the their tasks is a key issue. In this paper, we define the latest start time lst i j of each task, and use it to rank the workflow tasks. The lst i j of task t i j is defined as the latest time, before which t i j should be started; otherwise, the finish time ft i of workflow w i will be delayed and may miss its deadline. Definition 1: For task t i j , its latest start time lst i j is defined as: where succ(t i j ) represents a task set including all the immediate successor tasks of task t i j . Then, task t i j 's latest finish time lft i j is defined as: lft i j = lst i j + ct i j . Since tasks in a single workflow have precedence constraints, a large number of idle time slots will be left over in VMs when executing workflows. It is worth noting that there is no data dependency among tasks from different workflows. If tasks in different workflows are executed in proper sequences at the same VMs, the idle time slots in VMs can be reduced or even removed. Therefore, we can get the following principle. Principle 1: Assume task t i j and task t i j in workflow w i share the same VM vm kl , and the finish time of task t i j is less than the start time of task t i j , i. Proof: Let 1 be the idle time slot between task t i j and task t i j on VM vm kl , i.e., 1 = st i j ,kl − ft i j,kl . If task t i p is allocated between t i j and t i j , then the new idle time slot between t i j and t i j is 2 = st i j ,kl − ft i j,kl − et i p,kl < 1 , which proves that the idle time of VM vm kl is reduced.
Principle 1 means that tasks from different workflows executed alternately on the same VMs can reduce the idle time slots for VMs effectively.
Since a cloud data center contains massive heterogeneous physical hosts, each of which has different computation capacity and energy power. To reduce the energy consumption of a cloud data center, the following two issues should be addressed: (1) how can we minimize the total power consumption of all the active hosts in a cloud data center under a certain resource requirement; (2) which hosts should be turned on/off when scaling up/down the computing resources. For the first issue, we give the Principle 2.
Principle 2: Let H a = {h 1 , h 2 , · · · , h |H a | } be the set of active hosts, and the total CPU frequency of host set H a is supposed to be a fixed value, i.e., F = |H a | k=1 f k , where f k is a variable representing the frequency of host h k ∈ H a . The total energy power of host set H a , denoted as Proof: Based on Eq. (1), the power p k of active host h k can be rewritten as: Define the Lagrange function of formula (11) as: where f = {f 1 , f 2 , · · · , f |H a | } is a vector, and λ is a constant. Its optimal conditions are: (13) Thus, if and only if c 1 ·f 2 1 = c 2 ·f 2 2 = · · · = c |H a | ·f 2 k = − λ 3 , the total power h k=1 p k is minimized. Note that at one moment, the number of virtual machines in the data center is fixed, and the CPU frequency of each virtual machine is also fixed. Then, the total frequency that the data center needs to provide is fixed. This total CPU frequency is the assumption in Principle 2.
Principle 2 suggests that making c k · f 2 k as balanced as possible among all the active hosts in a data center is able to minimize its total power.

B. PROPOSED ALGORITHM
Definition 2: Task t i j is defined as ready if it has no predecessor task, i.e., pred(t i j ) = ∅; or all its predecessors have been scheduled to VMs.
On the basis of the scheduling architecture in Fig. 1 and Principle 1 and Principle 2, we develop an online workflow scheduling algorithm, NEAT, to deploy dynamic workflows with deadlines for execution in cloud data centers, while dynamically adjusting the available VMs and active hosts to achieve energy saving. Regarding traditional scheduling algorithms, once workflow tasks are scheduled, they will be dispatched to VMs or hosts immediately. Unlike those algorithms, NEAT only allocates workflow tasks that are ready to be executed to VMs, and keeps all the other waiting tasks in a task pool and their scheduling decisions can be reproduced to improve the resource and energy efficiency. The pseudocode of NEAT is shown in Algorithm 1.  18 selectedVM ← ScaleUpResources(); 19 Schedule task t i j to selVM ;

Algorithm 1 Pseudocode of NEAT
The proposed NEAT in Algorithm 1 works in a reactive fashion, once a new workflow arrives, it will be triggered to schedule all the new workflow tasks and the waiting workflow tasks in the task pool. The specific steps are as follows. Once a new workflow arrives the cloud data center (Line 2), the scheduling decisions that were previously generated and not executed will be cancelled (Line 3). The scheduling decisions include the mappings of waiting tasks in taskPool to VMs and the plan of scaling up/down resources. Then, the status information of all the VMs will be updated (Line 4). Before adding into the taskPool (Line 6), each task in the new workflow will obtain a weight (i.e., lst i j ) (Line 5). After that, all the tasks in taskPool will be scheduled to VMs in a hybrid way to improve resource and energy efficiency (Lines 7-19). Particularly, in each scheduling iteration, all the ready tasks from different workflows in taskPool are selected and sorted (Lines 4-5). To conserve energy consumption, for each ready task t i j , an initiated VM that can meet its resource requirement (Line 12) and complete it before its latest finish time lft i j with the minimum wasted resources wr i j,k,l is selected for task t i j (Lines 13-16). If the strategy above cannot select feasible VM (Line 17), the function ScaleUpResources() will be called to add a new VM (Line 18).
The pseudocode for function ScaleUpResources() is detailed in Algorithm 2. 1 Select a VM vm l whose CPU freq. is just larger than cpu i j ; 2 AH ← active hosts with unused CPU freq. larger than f l ; 3 destHost ← NULL; minValue ← +∞;

Algorithm 2 Function ScaleUpResources()
Initiate the VM vm k,l on destHost; return vm k,l ; 9 else 10 OH ← all the off hosts in the data center; Function ScaleUpResources() in Algorithm 2 firstly chooses a VM template vm l that meets task t i j 's CPU frequency requirement (Line 1). To reduce the total energy power according to Principle 2, the host h k that can accommodate the VM template vm l and has the minimal c k · f 2 k is chosen to deploy the new VM with template vm l (Lines 2-6). If the above mentioned step is infeasible, an off host h k with the minimal energy-frequency ratio (i.e., p max k /f max k ) that can contain vm l is turned on for the new VM (Lines 9-15).  + N h )) to schedule all the tasks in workflow w i . In summary, the time complexity of scheduling a workflow set W is calculated as O(|W |· max Definition 3: Idle VM: a VM is defined as idle if it has completed all the tasks mapped to it.
Since a wide range of end-users submit workflow applications to cloud platforms dynamically, their workload certainly fluctuate sharply over the time. When the local peak workloads over, there always appear some idle VMs and hosts in a cloud data center. In order to improve resource and energy efficiency, we devise a scale down strategy to reduce the number of used hosts by consolidating active VMs dynamically, which is outlined in Algorithm 3.  15 Execute the migration plan migPlan; 16 Turn off host h k , and remove it from AH ;

Algorithm 3 Function ScaleDownResources()
If the system workload becomes light and there exist some VMs that are idle. These idle VMs will be deleted, and idle hosts will also be shut down if all the VMs on them are deleted. Then, the function ScaleDownResources(), as shown in Algorithm 3, will be called to reduce the count of used VOLUME 8, 2020 hosts by consolidating the VMs and turning off the idle hosts. Firstly, all the active hosts are sorted by their f max k /p max k in an increasing order (Line 2). After that, this function starts from the active hosts with lower f max k /p max k (Line 3), and then checks if the VMs on the selected host can be migrated to other hosts (Lines 5-13). Moreover, to improve system's energy efficiency, this function strives to maintain the balance of c p · f 2 p of all the active hosts in canH (Lines 8-9). Notably, if there exist some virtual machines on the selected host cannot find any target hosts, the plan of migrating other VMs for the selected host will be given up (Line 13). Otherwise, migrate all the VMs on selected host to corresponding target hosts, and then turn off the selected host after migrations (Lines 14-16).

V. EXPERIMENTS
In this section, we conduct comparative experiments to verify the superiority of algorithm NEAT. These comparing algorithms are: EASA [23], HEFT [14], and ESFS [24]. EASA: it models a workflow as a set of precedence constrained virtual machines (VMs), and each VM represents a workflow task. Algorithm EASA firstly sorts all the VMs in a decreasing order based on their distances to the exit VM, then schedules VMs to hosts so as to optimize both makespan and host utilization, after that EASA tries to relocate VMs on lowutilized active hosts to high-utilized active hosts.
HEFT: this algorithm firstly assigns priority to each task according to its maximal length to the exit task. Then, the task with higher priority will be preferentially scheduled to a host with minimal energy consumption.
ESFS: this algorithm first utilizes algorithm HEFT to build an initial schedule of workflow tasks, and then gradually adjusts hosts' CPU frequency for each workflow task with the goal of reducing energy consumption while guaranteeing the deadline constraint of a workflow.
In addition, the energy consumption (EC) as formula (8), resource utilization (RU) as formula (9) and the count of used hosts, are chosen to compare NEAT with these three algorithms.

A. EXPERIMENTAL SETUP
We choose ten types of real hosts 2 to simulate the host configurations in a cloud data center, and assume the count of hosts with each type to 1000. The main parameters of these hosts are shown in Table 1. The parameters Max Fre., p idle k , and 2 http://www.spec.org/power_ssj2008/results/  Four kinds of widely-used workflows (i.e., CyberShake, LIGO, SIPHT, and Montage ) are selected, and three different sizes of each workflow: type one (about 30 tasks), type two (about 50 tasks) and type three (about 100 tasks), are used to simulate the dynamic applications. The approximate structures of first three workflow applications are shown in Fig. 2. We can observe that these three kinds of workflows cover all the basic structural features in terms of components (oneto-one, one-to-many, many-to-one, and many-to-many). The resource requirements and execution time of the workflow tasks and data sizes among tasks of these workflow are available in Pegasus repository. 3 The arrival of these workflows is assumed to be subject to the Poisson distribution with parameter λ. In addition, the symbol, arrival rate = 1/λ, is utilized to control the average number of new arrival workflows per second.
In addition, the parameter deadlineBase is used to reflect workflows' timing requirements, and the workflow deadline becomes looser with the increase of deadlineBase.
For each group of experimental setting, each algorithm is tested 30 times independently. Then, the average, minimum and maximum of experimental results are presented.

B. PERFORMANCE IMPACT OF WORKFLOW DEADLINE
This group of experiments focus on studying the impact of worklfow deadline on the performance of the four algorithms. The parameter deadlineBase is increased from 1.2 to 2.0, and the arrival rate and the number of workflows are set to 0.5 and 10,000. The comparing results are reported in Fig. 3.
As shown in Fig. 3(a), the energy consumption (EC) of the four algorithms remain basically the same regardless of the change of deadlineBase. In term of this metric, the proposed algorithm NEAT outperforms EASA, HEFT and ESFS by around 44.92%, 40.02% and 34.79%, respectively. The reason is threefold: 1) algorithm NEAT leverages the virtualization technology to simultaneously deploy multiple workflow tasks for execution on a host, so the count of used hosts can be reduced; 2) the fact that there exists no data dependence among tasks from different is exploited by NEAT to compress the idle time slots; 3) NEAT also dynamically adjusts the available VMs and hosts on the basis of the workload. Although algorithm EASA has considered the case that a single task cannot fully utilize the host resources, it always consumes the most energy. The reason is that it does not consider the heterogeneity of hosts when scheduling workflow tasks, and turns on hosts in a random way when increasing hosts. In addition, NEAT consumes less energy than EASA, HEFT and ESFS on average by 45.64%, 39.23% and 35.98%, respectively. This experimental result indicates that NEAT is efficient with energy conservation for executing workflows in virtualized cloud data centers.
The resource utilization (RU) of active hosts is shown in Fig. 3(b). With the increase of deadlineBase, the RUs of algorithms NEAT, EASA, HEFT and ESFS fluctuate slightly around 69.32%, 53.14%, 37.06% and 29.08%, respectively. The interpretation for such high RU of the proposed NEAT is that it schedules workflow tasks coming from different workflows in a hybrid way to reduce the idle time slots on resources, and also adjusts the available resources dynamically.
According to Fig. 3(a) and Fig. 3(b), we find an unusual phenomenon that EASA produces much higher resource utilization than HEFT and ESFS, but EASA still consumes much more energy than them, which seems contrary to the common sense that the higher resource utilization means the lower energy consumption. This can be explained by the fact that EASA does not consider the host heterogeneity in cloud data centers, and does not turn on/off the hosts with high/low energy efficiency when scaling up/down computing resources.
To compare the performance of the four algorithms with respect to the number of used hosts, we fix the workflow count, arrival rate, and deadlineBase as 10000, 0.5 and 1.2 to observe the number distribution of the used hosts for the four algorithms over time, which is illustrated in Fig. 3(c). This experimental result shows that the number of used hosts of NEAT is far lower than that of the other three algorithms, especially about 50% lower than that ESFS. The explanation for this result can be referred to that of Fig. 3(a).

C. PERFORMANCE IMPACT OF ARRIVAL RATE
In this group of experiments, the arrival rate is increased from 0.1 to 0.5 with an increment of 0.1. The workflow count and deadlineBase are fixed to 10,000 and 1.8, respectively. Fig. 4 illustrates the comparing results of the four algorithms NEAT, EASA, HEFT and ESFS. Fig. 4(a) reveals that the EC of the four tested algorithms increase slightly with the increase of arrival rate. The explanation for the phenomenon is that the higher arrival rate of workflows means the cloud data center needs to execute more workflows in each unit time, which requires more hosts, resulting in the use of more hosts with low energy efficiency. Besides, the advantage of the proposed NEAT is consistent with that in Fig. 3(a). We can also observe that NEAT consumes the least energy consumption, whereas EASA is the most energy consumption. This dues to that EASA does not consider the heterogeneity of hosts in the data center. In addition, NEAT consumes less energy than EASA, HEFT and ESFS on average by 45.60%, 36.83% and 34.00%, respectively. The above result is consistent with that in Fig. 3(a). Fig. 4(b) shows that the RU of NEAT decreases from 74.79% to 70.16% with arrival rate increases from 0.1 to 0.5. This can be explained by that with the increase of arrival rate, a cloud data center has to start more hosts, which also leads to higher heterogeneity of used hosts, thereby increasing the difficulties in scaling up/down computing resources. On average, NEAT outperforms EASA, HEFT and ESFS by 26.27%, 49.95% and 59.78%, respectively. The reasons are similar to Fig. 3(b). Fig. 4(c) provides the count distribution of hosts used over time for algorithms NEAT, EASA, HEFT and ESFS, when the workflow count, arrival rate, and deadlineBase are set to be 10000, 0.5 and 1.8, respectively. Fig. 4(c) shows that NEAT  turns on less hosts than EASA, HEFT and ESFS roughly by 50%, 32% and 30%, respectively. This result demonstrates the advantage of the proposed NEAT in reducing the host count. Besides, the advantage of the proposed NEAT is consistent with that in Fig. 3(a). It can be found that the count of used hosts for the four algorithms are elastic according to the request demand. Comparing with other three algorithms, algorithm NEAT boots up the least hosts all the time. Fig. 4(c) also shows that NEAT turns on less hosts than EASA, HEFT and ESFS roughly by 50%, 32% and 30%, respectively.

D. PERFORMANCE IMPACT OF CCR
The parameter CCR of a workflow refers to the ratio of its computational time and its communication time [23]. In this subsection, we attempts to study the impact of CCR on the four algorithms, and vary the CCR 0.5 to 5.5. The experimental results are shown in Fig. 5. Fig. 5(a) shows that the EC of EASA, HEFT and ESFS increases when increasing the CCR. This result can be contributed to the fact that larger CCR means higher communication time among workflow taks, which further results in longer idle time slots on VMs. In addition, these three algorithms have not employ effective strategies to compress the idle time slots, resulting in more energy consumption. It is worth noting that the EC of NEAT basically keeps unchanged with the variance of the CCR. This can be contributed to the strategy in NEAT that it deploys tasks coming from different workflows to run on the same VMs alternately, which is helpful of decreasing the idle time slots. Fig. 5(b) gives the first impression that NEAT is far superior to EASA, HEFT and ESFS in terms of RU. This result again reveals the advantage of the proposed NEAT in improving RU while meeting workflows' timing requirements. Fig. 5(b) also depicts that when CCR increases, the resource utilizations of algorithm EASA, HEFT and ESFS decrease correspondingly. This is because with the increase of CCR, the idle time slots on computing resources become larger, thus the resource utilization shows a descending trend. However, the resource utilization of NEAT shows a slight upward trend with the increase of CCR. This phenomenon demonstrates that the strategy, that scheduling tasks from different workflows to the same VMs, of NEAT is effective with respect to resource utilization. Fig. 5(c) depicts the change of the count of used hosts with time, when the workflow count, arrival rate, deadlineBase and CCR are set to be 10000, 0.5, 1.8 and 0.5, respectively. It is evident that the performance of the proposed NEAT remains consistently superior to that of its competitors with respect to the count of hosts used. Fig. 5(c) clearly shows that the hosts used of these four algorithms can be roughly classified into three categories: the count of hosts used by NEAT fluctuates between 3000 and 4000, and that of EASA and HEFT are around 5200, while ESFS initiates up to 7000 hosts. This result once again demonstrates our proposed NEAT can effectively reduce the count of hosts used by cloud data centers. This experimental result is consistent with that in 3(c), and the explanation for it is similar to that for Fig. 3(c).

VI. CONCLUSION AND FUTURE WORK
The focus of this paper is on reducing energy consumption for cloud data centers while guaranteeing the deadlines of dynamic workflows. We first propose a novel scheduling architecture to translate dynamic scheduling into static scheduling, then derive two principles to address the workflow scheduling issues. After that, we propose the algorithm NEAT to improve the energy and resource efficiency for cloud platforms while meeting timing requirements of workflows. Finally, three groups of experiments on real-world workflow traces are conducted to validate the performance of the proposed NEAT. Compared with three existing algorithms, NEAT shows better performance in terms of energy consumption, resource utilization and the count of used hosts.
When executing data intensive applications in industrial clouds, the network congestion is inevitable, which can also evidently impact the platforms' performance. Thus, we intend to develop novel contention-aware scheduling approaches for data-intensive applications in the future. Besides, it is reported that task execution times and data transmission times in cloud environments appear significant uncertainties, which may severely shake the performance of the algorithms. Thus another future study is to extend the NEAT to handle the uncertainties of task execution times and data transmission times.