A Heuristic Big Data Processing Multi Task Efficient Deployment Method Based on QoS Aware Clustering and Bayesian Classification in Cloud Environment

The efficient deployment of Big Data processing tasks in cloud environments is the basic core function of Big Data processing, which refers to the effective deployment of tasks to the computing resources of cloud platforms, achieving high-performance and high-throughput data processing. In this process, task deployment needs to consider load balancing on the cloud platform to ensure that tasks can be evenly deployed to each computing node. However, currently in the process of providing services on cloud platforms, the available resources of all hosts will be automatically and dynamically readjusted, and it cannot be guaranteed that each task will be deployed to the host with the most remaining resources. This load imbalance in the platform will result in computational results that cannot be returned to users in a timely and effective manner. So, a heuristic multi-task efficient deployment approach for Big Data processing based on QoS awareness and Bayesian classification in cloud environments called QBC is proposed. The QBC first performs long-run QoS awareness on hosts in the cloud; Then, based on user task requirements, selects host nodes that meet QoS constraints to form a candidate set, and performs Bayesian classification to find the host node which has highest a posteriori probability to serve as the clustering center; thirdly, designs an objective function based on Euclidean spatial distance to acquire the optimum host clustering set in the candidate set; Finally, deploys the user's tasks to this optimal host cluster set. The experimental results show that this approach implements optimization of long-run load balancing in Big Data cloud platforms with minimal resource consumption, enhances the ability of the cloud platform to provide external support, and thus promotes efficient deployment of multitasking in Big Data processing under cloud computing. The proposed QBC based framework reduces the overall Energy Consumption by an average of 47.98%, MakeSpan by an average of 24.42%, Total Cost by an average of 30.17%, Average Waiting Time by an average of 36.92%, and the Throughput is increased by an average of 41.93% as compared to the existing algorithms.