Predicting Inter-Data-Center Network Traffic Using Elephant Flow and Sublink Information

With the ever increasing number of large scale Internet applications, inter-data-center (inter-DC) data transfers are becoming more and more common. Traditional inter-DC transfers suffer from both low utilization and congestion, and traffic prediction is an important method to optimize these transfers. Inter-DC traffic is harder to predict than many other types of network traffic because it is dominated by a few large applications. We propose a model that significantly reduces the prediction errors. In our model, we combine wavelet transform with artificial neural network to improve prediction accuracy. Specifically, we explicitly add information of sublink traffic and elephant flows, the least predictable yet dominating traffic in inter-DC network, into our prediction model. To reduce the amount of monitoring overhead for the elephant flow information, we add interpolation to fill in the unknown values in the elephant flows. We demonstrate that we can reduce prediction errors over existing methods by 5%~30%. Our prediction is in production as part of the traffic scheduling system at Baidu, one of the largest Internet companies in China, helping to reduce the peak network bandwidth.