Deadline-Aware Scheduling and Flexible Bandwidth Allocation for Big-Data Transfers

Big data is becoming a major focus for both industry and academia, requiring drastic changes in all aspects of computer systems in order to store, process, and transfer big data. In networks, a fundamental problem is how to efficiently transfer big data since the performance is affected by several factors such as path, bandwidth, and scheduled start time. Best-effort algorithms are no longer applicable as they may not satisfy the deadline requirement of the requests. In this paper, we consider the problem of scheduling and flexible bandwidth allocation for big-data transfers with deadline constraints. With flexible bandwidth allocation, the bandwidth allocated to a request can be dynamically adjusted any time during its transfer. We develop an optimization programming formulation that provides admission and scheduling decisions, bandwidth allocation, and path selection for each accepted request. The formulation aims at maximizing the acceptance while guaranteeing the deadline constraints of transfer requests. Due to the complex nature of the optimization problem, we develop a two-phase heuristic algorithm namely deadline-aware flexible bandwidth allocation for big-data transfers (DaFBA). We develop two scheduling approaches for DaFBA using batch scheduling to be used for every time interval and dynamic scheduling to be used upon every request arrival. We evaluate the performance of the proposed algorithm through comprehensive simulations with two routing scenarios: pre-computed path scenario and load-based routing scenario. The results show that the proposed algorithm performs close to the optimal solution and outperforms baseline algorithms in terms of rejection ratio and the amount of data transferred.