Efficient Task Scheduling in Apache Mesos using Hybrid Stochastic Gradient Descent Golden Eagle Optimized (HSGD-GEO) Apache Aurora Framework

As the data centres expands rapidly must ensure that resource management software correlates closely with the hardware architecture to achieve optimal utilisation, performance, fault tolerance, and availability as they expand in size. Apache Mesos has become a prominent figure in this field by offering an abstraction that covers the whole cluster, data centre, or cloud to provide an accurate representation of all resources. Frameworks oversee organising and supervising tasks inside the Mesos cluster. They sign up with the Mesos master to get resource offers, which consist of resources that are accessible via Mesos agents. Frameworks have a scheduler component that determines how to efficiently use available resources according to their scheduling rules and the tasks that need to be executed. The Mesos agent, or slave, operates on each Mesos cluster node. The agent contacts the Mesos master to supply resources and receive tasks. Our proposed approach optimises Apache Aurora, an established framework that communicates with Mesos via its API for task scheduling, to optimise Apache Aurora, we designed the Hybrid Stochastic Gradient Descent Golden Eagle Optimisation algorithm (HSGD-GEOA). Apache Aurora dynamically adjusts task scheduling based on workload and available resources. Inspired by Stochastic Gradient Descent (SGD) methods, it adapts scheduling strategies using real-time inputs like workload patterns, resource utilization, and performance indicators. Using the GEO technique, the cluster dynamically assigns resources to meet workload demands, optimizing performance metrics such as response time, throughput, and energy efficiency, akin to the adaptive behaviour of golden eagles. The simulation results showed that the HSGD-GEOA method outperformed the other Apache Mesos Frameworks in terms of resource utilization, load balancing, and job execution perfomance.