Abstract:
Approximation jobs that allow partial execution of their many tasks to achieve valuable results have played an important role in today’s large-scale data analytics. This ...Show MoreMetadata
Abstract:
Approximation jobs that allow partial execution of their many tasks to achieve valuable results have played an important role in today’s large-scale data analytics. This fact can be utilized to maximize the system utility of a big data computing cluster by choosing proper tasks in scheduling for each approximation job. A fundamental challenge herein, however, is that the machine service capacity may fluctuate substantially during a job’s lifetime, which makes it difficult to assign valuable tasks to well-performing machines. In addition, the cluster scheduler needs to make online scheduling decisions without knowing future job arrivals according to machine availabilities. In this paper, we tackle this online resource allocation problem for approximation jobs in parallel computing clusters. In particular, we model a cluster with heterogeneous machines as a multi-armed bandit where each machine is treated as an arm. By making estimations on machine service rates while balancing the exploration-exploitation trade-off, we design an efficient online resource allocation algorithm from a bandit perspective. The proposed algorithm extends existing online convex optimization techniques and yields a sublinear regret bound. Moreover, we also examine the performance of the proposed algorithm via extensive trace-driven simulations and demonstrate that it outperforms the baselines substantially.
Published in: IEEE/ACM Transactions on Networking ( Volume: 28, Issue: 5, October 2020)
Funding Agency:
Keywords assist with retrieval of results and provide a means to discovering other relevant content. Learn more.
- IEEE Keywords
- Index Terms
- Resource Allocation ,
- Online Resource Allocation ,
- Efficient Algorithm ,
- Service Rate ,
- Scheduling Decisions ,
- Multi-armed Bandit ,
- Optimization Problem ,
- Upper Bound ,
- Random Sampling ,
- Computational Cost ,
- Gradient Descent ,
- Resource Constraints ,
- Feasible Solution ,
- Lagrange Multiplier ,
- Amount Of Work ,
- Time Slot ,
- Concave Function ,
- Constraint Violation ,
- Scheduling Algorithm ,
- Lipschitz Function ,
- Upper Confidence Bound ,
- Migration Cost ,
- Dual Variables ,
- Online Optimization ,
- Bandit Problem ,
- Online Algorithm ,
- Gamma Distribution ,
- Online Decision ,
- Online Manner ,
- Gradient Descent Method
- Author Keywords
Keywords assist with retrieval of results and provide a means to discovering other relevant content. Learn more.
- IEEE Keywords
- Index Terms
- Resource Allocation ,
- Online Resource Allocation ,
- Efficient Algorithm ,
- Service Rate ,
- Scheduling Decisions ,
- Multi-armed Bandit ,
- Optimization Problem ,
- Upper Bound ,
- Random Sampling ,
- Computational Cost ,
- Gradient Descent ,
- Resource Constraints ,
- Feasible Solution ,
- Lagrange Multiplier ,
- Amount Of Work ,
- Time Slot ,
- Concave Function ,
- Constraint Violation ,
- Scheduling Algorithm ,
- Lipschitz Function ,
- Upper Confidence Bound ,
- Migration Cost ,
- Dual Variables ,
- Online Optimization ,
- Bandit Problem ,
- Online Algorithm ,
- Gamma Distribution ,
- Online Decision ,
- Online Manner ,
- Gradient Descent Method
- Author Keywords