A resource scheduling strategy based on firefly algorithm

The Yarn scheduler is one of the core modules of Hadoop. It plays a critical role in the resource utilization efficiency and overall performance of a distributed cluster. However, with the generation of large amounts of data, the three resource schedulers it comes with can no longer well meet the needs of the industry. Therefore, this paper proposes a resource scheduling strategy based on the Firefly algorithm. Firstly, through the study of the scheduling of existing resources and cluster resource allocation mechanism, the algorithm based on the capacity scheduler is designed to achieve a more reasonable resource allocation scheme. Secondly, according to the job resource demand and the resource status of system nodes to dynamically adjust, we design the FireFlyScheduler resource scheduling strategy based on the Firefly algorithm. Finally, in order to test the effectiveness of the proposed scheduling strategy, the cultural data is tested with WordCount instances and two different types of MapReduce tasks TeraSort and TestDFSIO.Simulation results show that the performance of the proposed FireFlyScheduler scheduling strategy is better than that of other existing capacity schedulers. The above results prove the superiority of the scheduling strategy proposed in this paper, which is of great significance for improving the overall resource utilization of the system and the overall performance of the Hadoop platform.