An In-Depth Empirical Investigation of State-of-the-Art Scheduling Approaches for Cloud Computing

Recently, Cloud computing has emerged as one of the widely used platforms to provide compute, storage and analytics services to end-users and organizations on a pay-as-you-use basis, with high agility, availability, scalability, and resiliency. This enables individuals and organizations to have access to a large pool of high processing resources without the need for establishing a high-performance computing (HPC) platform. From the past few years, task scheduling in Cloud computing is reckoned as eminent recourse for researchers. However, task scheduling is considered an NP-hard problem. In this research work, we investigate and empirically compare some of the most prominent state-of-the-art scheduling heuristics in terms of Makespan, Average resource utilization (ARUR), Throughput, and Energy consumption. The comparison is then extended by evaluating the approaches in terms of individual VM level load imbalance. After extensive simulation, the comparative analysis has revealed that Task Aware Scheduling Algorithm (TASA) and Proactive Simulation-based Scheduling and Load Balancing (PSSLB) outperformed as compared to the rest of the approaches and seems to be optimal choice keeping in view the trade-of between the complexities involved and the performance achieved concerning Makespan, Throughput, resource utilization, and Energy consumption.


I. INTRODUCTION
For the last few years, task scheduling in Cloud computing got attention from the research community. The Cloud computing platform is deemed to provide high processing and huge memory-intensive services to the users in pay as you use manner. The resources in the Cloud computing are distributed across different locations in the form of large-scale data centers connected in a distributed way. In the Cloud computing environment, the resources are provisioned (to the users in an on-demand fashion) in the form of virtual machines (VMs) from a pool of configured resources (Physical Hosts (PHs)) keeping in view the requirements of the The associate editor coordinating the review of this manuscript and approving it for publication was P. K. Gupta.
intended users [1], [2]. The Cloud service owners provide guaranteed services called Quality of Service (QoS) to the users to meet the requirements of their customers [3]. These QoS contracts are formally maintained in the form of Service Level Agreements (SLA). To meet the SLA requirements of the users, the resources are provisioned aggressively from the Cloud Data Centers(CDCs) [4], [5]. Allocating resources in this way may lead to in-efficient resource allocation which will have a devastating impact on resource utilization across the Cloud. Moreover, will lead to workload imbalance across the CDCs causing data and computing skewness problem [6]. This will further result in under-utilization of some of the resources and on the other hand the waiting time of some of the jobs will also be increased [7]. To cope with these issues, the VM Migrations approaches are proposed. The VM migration approaches are useful in achieving load balancing, fault tolerance, scheduling optimization, and energy-aware resource/task scheduling. VM migration [8], [9] provides the ability to reduce the downtime pertaining to the overloaded VMs. With all its benefits, VM migration suffers from several issues including overhead involved and memory consumed during the migration phase. To mitigate these issues, a number of approaches are being introduced for VM migration. However, the required efficiency will not be achieved until and unless the VM migration is integrated with an efficient approach. One such approach will be to provide a load-balanced task distribution across the Cloud.
In the recent few years, the researchers have focused on introducing Cloud scheduling heuristics focusing on load-balanced provisioning of tasks with the aim of producing efficient resource utilization. The Min-Min [11] scheduling algorithm was introduced to give priority to the tasks with a smaller size. However, the tasks having larger size had to wait for longer times before executing them on the available resources thus leading to inefficient resource utilization. In [10], Chen et al. extended the existing Min-Min [11] approach by the proposal of two efficient load balancing heuristics (namely Load-Balanced Improved Min-Min (LBIMM) and User Priority-aware Load-Balanced Improved Min-Min (PA-LBIMM)). The PA-LBIMM considers the users' priority while keeping the load balanced scheduling thus provide improved results in terms of achieving the required level of SLA and resource utilization. This approach also assures a gain in the required level of throughput, better resource utilization, and better response time for all the required tasks. The authors in [12] introduced and implemented a RePro-active scheduling framework that utilizes the existing knowledge related to the tasks with the aim of load-balanced scheduling. The Max-Average scheduling heuristics was introduced that extended and improved the Max-Min algorithm with an aim of providing load balanced task scheduling across the CDCs [13]. Chauhan and Joshi proposed two different QoS aware heuristics (i.e., QWMTS (QoS Weighted Mean Time Min-Min Max-Min Selective) and QWMTM (QoS Guided Weighted Mean Time-Min)) for load-balanced task scheduling across the CDCs. The authors in [14] designed and implemented a load-balanced algorithm by extending the Max-Min strategy that keeps the status of each of the tasks. This information is used as an input for predicting the real-time load on each of the VM across the Cloud. The SLA-Min-Min and SLA-MCT approaches are presented in [7] that consider the cost and task execution time as the QoS parameters for guaranteed scheduling.
The task scheduling in the Cloud computing is NP-hard in nature. This means a single metric does not guarantee the efficiency and effectiveness of any task scheduling approach. Several metrics are utilized to measure the efficiency and optimality of the task scheduling approaches. Each evaluation metric measures the performance of the task scheduling approaches from a different perspective. Thus, it is not trivial to decide which task scheduling approach is the optimal choice. Keeping in view these points, in this research, we have selected few renowned state-of-the-art static task scheduling heuristics (i.e., task aware, resourceaware, and hybrid approaches) for an in-depth empirical investigation concerning five performance metrics (i.e., ARUR, Makespan, Throughput, Energy consumption, and Individual level VM load imbalance). In order to quantify the performance of each of the state-of-the-art approaches (compared in this work), several parameters are utilized as reported in Table 1. The Table provides three types of information that are: heuristic name, parameters used for evaluating the performance, and one main parameter that meets the objectives of the proposed scheduling approaches. In order to see the in-depth analysis of the proposed approaches, several approaches (i.e., [19], [21]) have provided a comparative analysis of the available contemporary approaches by employing the parameters given in Table 1. Moreover, it is observed that most of the approaches considered makespan and ARUR as the performance metrics to measure the load balancing and resource utilization attained by the scheduling approaches which are measured as: where CT j shows completion time of VM j and m represent number of VMs where avgMakespan is computed in Equation 3 The throughput can be measured as: From the comparative analysis of the state-of-the-art scheduling approaches (provided in various studies), it is observed that a higher value of ARUR is achieved, however, there is a need to investigate and address the machine level load-imbalance for improving the profitability of the cloud resources. Moreover, the energy consumption also need to be investigated for all the compared approaches. The contributions of this work are presented as follows: Empirical evaluation and comparative analysis of state-of-the-art static task scheduling approach using one of the most prominent Cloud benchmarks Dataset (i.e., HCSP instances) implemented using CloudSim [20]. In the last portion of this work, we outlined few recommendations for the cloud service provides as well as for the cloud users based on the comparative analysis results in terms of machine-level load-balancing, energy consumption, and corresponding resource utilization.
The rest of this paper is organized as follows: Section II delineates the aspirations of this study to highlight the motivations of this work. In Section III, we discuss the task scheduling algorithms working and corresponding parameters utilized for performance investigation. Section IV provides the details regarding experimental configuration setup, Dataset selection, and obtained results. The Result discussion and recommendations are delineated in Section V and finally, Section VI reports the conclusions and future work of this study.

II. OBJECTIVES
Task scheduling and load balancing have become one of the most important and attractive areas of research in the Cloud computing domain. A number of Cloud task scheduling heuristics have been proposed by different researchers. These scheduling heuristics have different scheduling objectives which include minimization of makespan, improving utilization of Cloud resources, SLA, and enhancement of throughput in the Cloud. Some of these approaches focus on reducing makespan while other approaches focus on improving resource utilization and/or throughput. However, these tasks scheduling algorithms have various levels of resource utilization, throughput, and makespan [22]. Moreover, various researchers have used different datasets, different number and combination of VMs with respect to their computation capabilities. This arise the need to comprehensively investigate and empirically examine state-of-the-art tasks scheduling heuristics using available datasets in terms of tasks heterogeneity and computation requirements, and VMs computation capability. The aim of this research is to see the effectiveness of state-of-the-art Cloud tasks scheduling heuristics in terms of makespan, ARUR, Throughput, and energy consumption. Some of the researchers have used homogeneous workload and Cloud environment and others have utilized heterogeneous Cloud resources and workload. However, in a real Cloud environment, both workload and computation resources are heterogeneous in nature. Therefore, we have used four sets of HCSP datasets each of which has workload with a different number of tasks and level of heterogeneity. Similarly, Virtual Machines (VMs) with different computation capabilities have been used. The objective of using different datasets and computing resources is to see how each of the algorithms behave with different datasets. This research will also highlight the load imbalance for each individual VM on various datasets. Moreover, we will identify the maximum, minimum, average load-imbalance behavior of each of the tasks scheduling heuristics.
For this, the computation share of each VM needs to find out which is calculated by using VMs computation power in Million Instructions per Second (MIPS) and total computation requirements of workload using equation 5.
where Clt i shows computation requirements of Cloudlet i in terms of Million Instructions (MIs), VM k and VM j represent computation power of VM j and VM k . Comp-Share j corresponds to the computation share of VM j . The difference between the computation share of VM and the load assigned by different algorithms is identified and categorized as load-imbalance.

III. TASK SCHEDULING APPROACHES
This section of the study discusses and critically investigate the working of some of the most prominent state-of-the-art static task scheduling algorithms as following. Minimum Completion Time (MCT) [15] based scheduling heuristics select tasks in the given order and assign them to the VMs that execute the task in minimum time. For this, the MCT based approach scans all VMs for each task and calculate their completion time. On each scheduling decision, VM load/ready time is updated. MCT can reduce makespan but overload faster VMs and slower VMs remain idle. This approach results in poor resource utilization.
Min-Min [11] tasks scheduling heuristics uses basic principles of MCT for tasks to VM mapping. Min-Min scheduling heuristics calculate Expected Execution Time (EET) for all tasks on each of the VM. The Min-Min algorithm selects the task with minimum EET and assigns the task to the resource which executes that task in minimum time. After mapping a task on VM, the load of that VM is updated and the mapped task is removed from the task list. Min-Min based scheduling heuristic favors smaller tasks and penalize tasks with the larger size. This may result in reduced resource utilization and throughput.
Max-Min [16] scheduling heuristic is an MCT-based approach. This approach first calculates the execution time of all tasks on each VM and then selects the row with the highest execution time. In the already selected row, the Max-Min approach selects VM with minimum execution time and map tasks on that VM. On each scheduling decision, the VM load is updated. Tasks execution is started after mapping all the tasks on the respective VMs. Max-Min based scheduling heuristics favor larger tasks and penalizing smaller tasks.
Max-Average (MaxAvg) is Max-Min based improved task scheduling heuristic proposed in [13] and complete their scheduling in two phases. In the first phase, the proposed approach uses the Max-Min approach to calculate expected completion time for a set of tasks on each VM. This approach selects the largest task and identifies the VM(s) that execute the selected task in Minimum Completion Time (MCT). In case, there is more than one candidate VMs that can execute the task with MCT, the proposed approach will assign the task to the resource with the least usage. To identify the resource with the least usage, it calculates the average completion time of all the resources. However, if average completion time is less than the MCT of the smallest task, then it selects the largest task and assigns the task to VM that executes them in minimum time. Otherwise, select the smallest tasks and assign the task to VM with MCT. This scheduling heuristic performs better for smaller datasets while high makespan and reduces throughput for larger tasks. Min-Min [11] scheduling heuristic reduces the makespan as compared to other tasks scheduling heuristics. However, the main issue with the Min-Min algorithm is poor resource utilization which is one of the key requirements of Cloud Service Provider (CSP). Authors in [10] have proposed an improved load balancing scheduling approach namely Load Balanced Improved Min-Min (LBIMM). This algorithm is based on Min-Min task scheduling heuristics. Moreover, this approach also can schedule users' jobs according to their priority like job execution time and cost. The proposed approach provides a user with the option to select the type of services they need. To evaluate the proposed technique parameters like makespan, resource utilization and average completion time of high priority jobs are considered.
PSSLB is a proactive simulation-based scheduling and load balancing algorithm proposed in [12]. This algorithm sorts a batch of incoming tasks in descending order of their size and calculates completion time of every task on all the available resources (i.e., VMs). PSSLB scheduling heuristic store ETC (expected time completion) in the form of a matrix, where each row represents the completion time of a task on every VM. The heuristic then selects the tasks with minimum completion time.
The authors in [18] proposed Resource Aware Scheduling Algorithm (RASA). RASA exploits the strengths of Min-Min and Max-Min tasks scheduling heuristics. A set of tasks are provided as an input to RASA that calculates the execution time of every task on each VM and stores it in the form of a matrix. This approach uses Min-Min tasks scheduling heuristic if the number of tasks is odd. For an even number of tasks, this approach applies the Max-Min based approach to all remaining tasks. RASA provides a fair task allocation criteria for small as well as large size tasks. However, RASA leads to load imbalance if there are a higher number of large size tasks. Sufferage [23] task scheduling heuristic uses MCT as a base approach and calculate execution time for each task on the available resources (VMs). sufferage scheduling classifies MCT and second MCT (second least execution time) for each task and then determine sufferage value. Sufferage value is identified by finding the difference between two tasks i.e task with MCT (minimum completion time) and task with second MCT. Sufferage produces high makespan and lower throughput for larger datasets having a substantial number of larger tasks.
Task Aware Scheduling Heuristic (TASA) [24] works in two phases that are; In the first phase, TASA uses Min-Min scheduling heuristic which ultimately favors smaller jobs. In the second phase, sufferage task scheduling heuristic is used to select a suitable VM for the selected task.

IV. EXPERIMENTAL SETUP
To evaluate the performance of any scheduling heuristics for Cloud computing, three different approaches (i.e., experimental, analytical, and simulation) can be used. The experimental techniques are expensive and are difficult to set up and may require an expert to design the testbeds. Moreover, utilizing the real Cloud platform works on a pay as you use model, thus executing simulation multiple times may result in high monitory cost. Whereas, the analytical techniques are often limited in evaluating the proposed scheduling heuristics. For the last few decades, the simulation approaches are extensively used to evaluate the performance of the underlying scheduling approaches using a wide variety of configurations. To investigate the performance of the available contemporary approaches (chosen in this work), a renowned simulation platform Cloudsim [20]is utilized. The computing powers of VMs are assigned in ascending order of their VM-ID. The simulation experiments were executed on a workstation equipped with Intel Core i5-8500 Quad-core processor (3.0 GHz clock speed) and 8 GBs of main memory.

A. DATASET SELECTION
To analyze the performance of state-of-the-art heuristic algorithms, we chose the heterogeneous Computing Scheduling Platform (HCSP) dataset proposed by Braunt et al. [27]. This model is based on Expected Time to Compute (ETC) matrix with m number of tasks and n number of VMs. The instance with size 1024 x 32 is considered for the evaluation of the compared heuristic approaches [28]. Each dataset instance denotes 1024 tasks also called Cloudlets (tasks) and 32 virtual machines (VMs). Four different instances (i.e., c-hilo, i-hilo, c-lohi, and i-lohi) are utilized. These instances are based on task heterogeneity and resource heterogeneity as described below;  hilo: Heavy set of tasks with the light capacity of resources lohi: Light set of tasks and high capacity of resources The details regarding the utilized dataset is also available at [29]. In the case of hilo datasets, most of the Cloudlets are heavier and the available VM resources are lighter in terms of the available CPU power. Whereas, the lohi datasets contain Cloudlets having smaller sizes and the resources (VMs) are powerful in terms of the CPU MIPS [29]. Only a single data center is considered in the simulation that is comprised of 32 servers. Each of the server is equipped with a RAM of 2GB and CPU of power 10,000 MIPS. The VMs are provisioned resources from this available pool of the server resources. In Table 2, the details about the VMs CPU specification in MIPS (million instructions per second) and task size (required MIPS) for all of the four datasets are provided. Also, the details regarding each VMs' CPU power (in MIPS) are reported in Table 3.

B. EXPERIMENTAL EVALUATION
The effectiveness of the scheduling algorithms depends on a number of factors that are: Execution time (Makespan), the average resource utilization (ARUR), throughput, the level of SLA violation, etc. From the users' point of view, the execution time with minimum SLA violation is of primary concern. On the other hand, the service providers' focus is to obtain maximum profitability from their services. Thus to achieve this, there is a need to develop and deploy schedulers that can handle a large number of requests and efficiently utilize the resources. This work specifically targets these areas that can provide performance trade-off of different scheduling heuristics to enable the service providers for better decision making. The obtained results and their discussion is presented below.
The results in Figure 1 demonstrate the average resource utilization on all the contemporary scheduling approaches (i.e., LIBMM, MaxAvg, MaxMin, etc.) compared in this research. The obtained results reveal several important peculiarities. The PSSLB and TASA approach lead to higher ARUR (0.97 and 0.99) as compared to the other compared approaches. The TASA and PSSLB approaches have achieved 155 %, 223 %, 162 %, 10 times, 148 %, 155 %, 11 times and 157 %, 227 %, 167 %, 10 times, 150 %, 157 %, 11 times as compared to LIBMM, MaxAvg, MinMin, MCT, MaxMin, RASA, and Sufferage respectively for the c-hilo dataset. This improvement is due to the balanced task mapping over the available resources. Sufferage and MCT resulted in poor resource utilization by acheiving only 10 or less than 10 % average resource utilization. This behavior is due to the fact that MCT burdens the resources (VMs) that provide minimum completion time for the Cloudlets while slower VMs remain idle. Likewise MCT, the sufferage under-utilizes the slower VMs resulting in poor resource utilization. The attained ARUR also depends on the available tasks in the datasets. For c-hilo and i-hilo dataset, the PSSLB and TASA attained higher ARUR. However, the ARUR value is dropped (from 0.98 and 0.97 to 0.33 and 0.39 respectively) for the PSSLB on the c-lohi and i-lohi datasets. On the other hand, TASA exhibited consistent behavior for all the datasets utilized. This means that task nature also impacts the behavior of the scheduling approach used. From the given discussion, it is concluded that TASA provides more stable and scale-able performance as compared to PSSLB and other compared approaches.
The mapping of tasks on the available VMs in a proper way has a clear impact on the overall execution time for task scheduling. The more balanced the mapping of tasks is the less the overall makespan for the available Cloudlets (i.e., tasks). To obtain an insight into the performance of the available scheduling heuristics (i.e., LIBMM, MaxAvg, MaxMin etc) with respect to makespan the simulation is executed multiple times, by employing HSPC [28] datasets (i.e., c-hilo, c-lohi, i-hilo and i-lohi). The results regarding the attained makespan on each of the scheduling approach (utilized in this research) are reported in Figure 2. The obtained results show that all the algorithms lead to reduced makespan for i-lohi dataset. For all the reported makespan results, the worst value of makespan is observed for sufferage on all the available datasets. The TASA and PSSLB outperformed as compared to other approaches (utilized for comparison in this research) in terms of makespan. For the hilo dataset instances (i.e., c-hilo and i-hilo), the TASA and PSSLB shown similar performance and shown an improvement of 170 %, 160 %, 157 %, 6 times, 170 %, 140 %, 10times as compared to LIBMM, MaxAvg, MinMin, MCT, MaxMin, RASA, and Sufferage respectively.
The in-depth investigation shows that TASA been able to produce the best makespan (10,000 and 5000 seconds) for all the available datasets. This is due to the fact that TASA maps the tasks in a way that keeps all the resources busy evenly during the execution of tasks on the available resources. The obtained results assert that TASA can be considered as the fittest candidate for the Cloud service providers. This concludes that understanding the characteristics of tasks is important for the design and development of any scheduling heuristic.  The best case and worst case throughput results for all the compared scheduling heuristics on the available HSPC datasets are plotted in Figure 3. Higher throughput value shows better resource utilization across the available VMs. The obtained results show a substantial increase (minimum of 115 % and 160 % improvement against MCT and 9 times, 11 times improvement against sufferage) in throughput on h-lohi and i-lohi for the PSSLB and TASA of the available the contemporary scheduling approaches used in this study. The worst performance is observed for MCT and sufferage on the hilo datasets (i.e., c-hilo and i-hilo). However, higher throughput is observed for MCT on the lohi datasets as compared to all the approaches except PSSLB and TASA. In case of lohi dataset, most of the jobs are of small size. Therefore, the MCT map tasks to the faster machines thus leading to completing a higher number of tasks within shorter makespan. This means that for jobs with small size, the MCT will be able to satisfy the SLA in an ideal way. On the other hand, the PSSLB is ideal for datasets with larger size jobs. The obtained results reveal that TASA shows consistent behavior for all the datasets, achieving higher throughput for both hilo and lohi datasets.
For the last few years, the energy consumption has been considered a vital metric for evaluating the performance of any Cloud scheduling approach. The energy consumption has been completely ignored by most of the authors while investigating and comparing their proposed task scheduling approaches performance against other state-of-the-art approaches. Thus, in this study we also attempted to analyze and compare the performance of the contemporary task scheduling approaches considered for comparison in this study. The energy model used in [30] is considered for empirical investigation. The results regarding the energy consumption for all the task scheduling approaches are reported in Figure 4. The best and worst results for all the dataset instances are shown obtained and plotted in Figure 4. Again the TASA and PSSLB approaches outperformed against the rest of the compared approaches concerning the energy consumption and lead to an average improvement of 130 %, 110 %, 125 %, 190 %, 140 %, 110 %, 197 % as compared to LIBMM, MaxAvg, MinMin, MCT, MaxMin, RASA, and Sufferage respectively. For the hilo dataset instances, the minimum energy consumption observed is 0.052 and 0.054 for the TASA and PSSLB respectively while higher energy values are reported for the MCT and sufferage approaches. The MinMin, MaxAvg, and RASA have shown moderate performance concerning the energy consumption for both the hilo and lohi datasets. For MCT and sufferage, higher energy consumption is observed for both the datasets and this is due to the reason that these approaches overload some of the faster VMs while the slower VMs remain idle. In most of the studies, it is reported that an idle machine consumes 70% of its resources even with no process running on it. Thus, for MCT and sufferage, the energy cost of these idle VMs leads to an increase in the overall energy consumption.
The effectiveness and performance of any scheduling algorithm depend on the makespan and resource utilization. The resource utilization is directly affected by how the available tasks are mapped to the available resources. The more balance the share of the load on each of the resource the more likely is to maximize the resource utilization which ultimately leads to reduced makespan with the better response time.
To obtain an insight into the performance of each scheduling algorithm, a number of experiments are performed using four sets of HCSP datasets (i.e., h-hilo, h-lohi, i-hilo, i-lohi). The obtained results pertaining to the percentage individual machine level and average load imbalance for all the compared scheduling heuristics (used in this study) are reported in Table 4. The MCT and sufferage lead to a higher level of load imbalance (i.e., 49.89% and 47.66%), where the MCT and sufferage under-utilize the slower and faster machines respectively. LIBMM and RASA produced a moderate level of load-balanced scheduling that resulted in better resource utilization. The Min-Min and RASA results in overloading the faster VMS slightly while leading to a load imbalance on the slower VMs (i.e., ). The PSSLB and TASA produce very balanced scheduling on all the VMs with negligible load imbalance on a few VMs (i.e., vm5, vm19, and vm29 for PSSLB and vm0, vm3, vm5, etc. for TASA). This, in turn, results in high resource utilization as shown in Figure 2. From the obtained results it has reveled that PSSLB and TASA outperformed as compared to other compared approaches by producing almost a completely load-balanced schedule on the available VMs.
To examine the behavior of each scheduling algorithm, the experiments were extended and executed for c-lohi dataset. In lohi dataset, most of the tasks are of smaller size and the VMs are high processing. The simulation was then performed several times with different seed values and results were obtained for each of the scheduling approaches. The results regarding individual machine level load imbalance (percentage) and average load imbalance for all the heuristics are presented in Table 5. The best and worst-case performance achieved by each of the scheduling approaches in terms of load imbalance is shown as bold in the table. Again, the PSSLB and TASA shown consistent load-balanced mapping (i.e., on most of the VMs 0% overload/under-load and only few VMs less than 0.06% for TASA and 0.22% average load imbalance for PSSLB) of the tasks on the available resources and therefore, outperformed as compared to all of the contemporary approaches (considered for comparison in this paper). The MaxAvg and LIBMM overloaded 6 to 9% of the faster VMs whereas a slight under-utilization is observed for the rest of the VMs. Similarly, the MCT severely overloaded the faster machines while mapping zero percent of tasks to the slower VMs (i.e., VM0 to VM20). Likewise MCT, the sufferage overloaded 58% of the VMs while the remaining VMs were idle during the course of scheduling. This imbalance leads to the under-utilization of resources as can be seen in Figure 1. These results also support the use of PSSLB and TASA as good candidates for scheduling tasks with most of the tasks having a smaller size. VOLUME 8, 2020 To examine the performance of the available contemporary scheduling heuristics, another set of experiments was performed on i-hilo and i-lohi datasets with tasks having inconsistent heterogeneity. Similarly, the VMs utilized for the experiments are also inconsistent in terms of heterogeneity. Table 6 presents the best and worst-case results of individual VM level load imbalance and average load imbalance for all the compared scheduling heuristics. The PSSLB and TASA achieved almost balanced task distribution on all the available VMs. For PSSLB, 75% of the VMs got exactly the same share of workload as provided in Eq 5 that shows the distribution of tasks equally among all the VMs. On the other hand, TASA leads to an impressive average load balance scheduling (96%) of the tasks on the available resources. The Max-Avg slightly overloads some of the slower and faster VMs. However, it leads to an improvement of 57% in terms of average load imbalance. The worst-case load distribution is observed for MCT and sufferage. MCT overloads the faster VMs and sufferage overloads 33% and under-utilizes 24% of the VMs whereas the remaining VMs are idle during the course of workload execution. The LIBMM and RASA have shown a moderate load-balanced task distribution among the available resources (i.e., VMs). The obtained results advocate the use of PSSLB and TASA as top candidates for scheduling jobs across the Cloud data centers.
The final set of simulation experiments is executed using i-lohi dataset to empirically investigate the performance of the compared approaches. The results in terms of ARUR, Makespan, Throughput, and load imbalance are obtained where the results concerning the first three parameters are plotted in Figure 1, Figure 2, and Figure 3 respectively and discussed accordingly. Table 7 reports the results to appertain to the individual machine load imbalance and the average load imbalance. PSSLB and TASA have shown superior performance as compared to other state-of-the-art compared approaches (used in this study) by producing almost a complete load-balanced scheduling of tasks on the available resources. MCT and sufferage are unable to map the tasks on the available VMS in a load-balanced way thus resulted in a higher level of load imbalance. LIBMM, Maxmin, MinMin, and RASA resulted in a moderate level of load distribution.
All these results assert that PSSLB and TASA remarkable performance against all the compared contemporary approaches for all the datasets showing consistent performance in terms of ARUR, Makespan, Throughput, and load imbalance. More in-depth analysis explicates that TASA is the leading algorithm and is considered to be the most appropriate choice for a variety of tasks whether small size or large and for datasets either consistent heterogeneous or inconsistent heterogeneous.

V. RESULT DISCUSSION AND RECOMMENDATIONS
Following the experimental performance evaluation, the result discussion is delineated to extract important findings regarding the scheduling approaches and the nature of datasets. The behavior of a number of scheduling approaches (used in this study) was empirically investigated using a diverse set of large size HCSP heterogeneous datasets. Among the compared scheduling heuristics, the PSSLB and TASA have shown substantial performance with respect to ARUR, Makespan, and throughput. Moreover, these scheduling approaches lead to map the resources in a completely balanced way resulting in 97 to 99% resource utilization. Among PSSLB and TASA, the TASA has shown consistent performance for all the datasets; however, the performance in terms of attained ARUR of PSSLB dropped to 30% and 38% for the c-lohi and i-lohi datasets respectively. RASA has been able to perform well in terms of ARUR for the c-hilo dataset as compared to other scheduling heuristics except for PSSLB and TASA. MCT and sufferage lead to poor resource utilization due to the fact that MCT overloads faster VMs while slower VMs remain idle. Similarly, sufferage like MCT under-utilized some of the slower machines resulting in poor resource utilization causing high Makespan. The under-utilization of the resources also results in an increase in the overall energy consumption. LIBMM and MaxAvg overloaded 6 to 9% of the faster VMs whereas a slight under-utilization is observed for the rest of the VMs. The TASA and PSSLB has been able to dramatically reduce Makespan as well as the energy consumption and also resulted in almost completely load-balanced scheduling of the tasks over the available resources. The TASA approach has shown consistent performance for the diverse nature of utilized dataset instance and thus outperformed all the compared task scheduling approaches concerning the utilized performance metrics.
From the Cloud service providers point of view, the resource utilization is considered as a more crucial parameter for the selection of scheduling approach. Moreover, poor resource utilization may lead to a load imbalance with high energy consumption. The problem concerning poor resource utilization needs to be addressed in a careful manner. For efficient resource utilization, the task should be mapped to the available resources keeping in view the available computing resources, workload properties, and already mapped tasks to the resources. The resource and task aware mapping of cloudlets on the available VMs in a load-balanced manner will lead to eliminating the problem of under-utilized/overloaded VMs ultimately resulting in reducing higher energy costs.
The obtained results reveal several important facts about the available contemporary scheduling approaches. With similar ARUR values, different Scheduling approaches employ different workload distribution on the available resources. This behavior further affects the Makespan and Throughput achieved. LIBMM and RASA have attained 0.39 and 0.41 ARUR (5% improvement) respectively using the c-hilo dataset; however, the average load imbalance faced by LIBMM and RASA is 1.64 and 1.41 respectively (14.5% improvement). This means that achieving the same ARUR, does not mean that load imbalance can be the same. Similarly, the same trend of different load imbalance produced by the scheduling heuristics having almost similar results for ARUR is observed in the simulation experiments done using HCSP datasets. In summary, TASA scheduling heuristic outperformed and shown scale-able performance as compared to the available approaches for all the utilized HCSP datasets. After careful empirical investigations, this research recommends the following important points for the Cloud service providers and researchers working in the task scheduling area.
The execution time (Makespan) and resource utilization are affected by the level of load-balanced workload distribution among the available resources. The load-balanced mapping of tasks is possible only and only if the nature of tasks and resources is to understand properly. TASA, a task aware scheduling algorithm endorsed this fact by attained almost balanced load scheduling for all the datasets.
To distinguish between different heuristic approaches, a number of datasets with diverse nature should be used to evaluate their performance.
Moreover, Cloud service providers can expand their revenue-generating with balanced energy consumption in the Cloud data centers by load-balanced workload scheduling, smaller Maksespan, and efficient resource utilization.
The reduction in execution time can also lead to satisfying Cloud service users.

VI. CONCLUSIONS AND FUTURE WORK
This research work evaluated several renowned state-of-the-art approaches rigorously by employing one of the most prominent benchmark dataset instances(i.e., HCSP) and considered a widely used simulation platform namely Cloudsim for evaluation comparison. The workload used for the experi-ments has been configured with different heterogeneity of the tasks and VMs. Among the available compared approaches, PSSLB and TASA have been able to attain the highest resource utilization with minimal makespan for HCSP c-hilo and i-hilo datasets. It was shown that the ARUR of PSSLB reduced to 30% and 38% (from 99% for c-hilo and i-hilo datasets) for the c-lohi and i-lohi datasets respectively. On the other hand, it was demonstrated that TASA scheduling heuristic outperformed and scale-able performance as compared to the compared approaches for all the utilized HCSP datasets. Moreover, the TASA and PSSLB also resulted in a very low energy consumption as compared to the rest of the approaches. MCT and sufferage are unable to map the tasks on the available VMS in a load-balanced way thus resulted in a higher level of load imbalance resulting in an excessive energy cost. LIBMM, Maxmin, MinMin, and RASA resulted in a moderate level of load distribution. The TASA and PSSLB have shown substantial performance concerning the energy consumption for all the dataset instances.
As a future, we plan to compare the available contemporary scheduling approaches with respect to SLA violation. The future work also aims to propose a task cum resource-aware scheduling approach that will exploit the nature of the presented workload and efficiently map the tasks on the available Cloud resources.  His research interests include algorithms, semantic web, and optimization techniques. He focuses on enhancing real-world matching systems using machine learning and data analytics in a context of supporting decision-making. VOLUME 8, 2020 MUHAMMAD SUMMAIR RAZA received the Ph.D. degree from the National University of Science and Technology (NUST), in 2018. He is currently working as an Assistant Professor with the Computer Science Department. He has published various research papers in national/international level journals and conferences.
ALTAF HUSSAIN received the B.S. degree (Hons.) in computer science from NWFP AUP, Pakistan, and the M.S. degree in computer software engineering from the National University Science and Technology, Islamabad, Pakistan, in 2010. He is currently pursuing the Ph.D. degree with the Capital University of Science and Technology, Islamabad. He is working as the Head of Department with the Department of Computer Science, KICSIT Campus, IST, Islamabad, Pakistan. His research interests include software testing, data mining and distributing computing comprises performance analysis, and cloud computing.
KHALED SALAH (Senior Member, IEEE) is currently a Professor with the ECE Department, Khalifa University of Science, Technology, and Research (KUSTAR). He has over 170 publications in areas related to the IoT, blockchain, cybersecurity, and cloud computing. He is on editorial boards of a number of WOS-listed journals, including IET Networks, IET Communications, Wiley's SCN, Wiley's IJNM, Elsevier's JNCA, J.UCS, and AJSE.
KARIM DJEMAME (Member, IEEE) received the Ph.D. degree from the University of Glasgow, U.K., in 1999. He is currently a Professor with the School of Computing, University of Leeds. He sits on a number of international programme committees for cloud middleware, computer networks, and performance evaluation. He was the investigator of various e-science/cloud projects, including DAME, BROADEN, Assess-Grid, ISQoS, STRAPP, OPTIMIS, and ASCETiC. He is currently involved in various research projects, including TANGO. His main research interests include grid/cloud computing, system architectures, resource management, and energy efficiency. He is a member of the BCS.