INVA: An Intelligent Network Virtualization Architecture for Big Data Platform

Various big data ecosystems emerge as the "big data" grows rapidly in the past decade when traditional approaches turn impractical. Due to its unique computing paradigm, a soaring amount of east-west (machine-to-machine) traffic is generated inside the big data platform, which inevitably congests the underlying network and therefore degrades the overall performance of the infrastructure. A series of work explore the opportunity to optimize the underlying infrastructure for the hosted big data applications (Chen and Zhang 2014a), but few seek to leverage the "big data" generated from the infrastructure to optimize the big data platform itself.In this work, we propose an intelligent network virtualization architecture for big data platform, i.e., INVA. By collecting and tapping the abundant system data in realtime, INVA endows the network with intelligence that facilitates an agile and self-learning network substrate for big data applications. We deployed INVA on a commercial big data platform with production traffic, and spotted the interesting heavytail phenomena, based on which we design a practical traffic prediction mechanism. In addition, we also propose a randomized provisioning algorithm for virtual network functions (VNFs) with theoretical bound. The experiment results show that INVA significantly boosts the performance of the big data platform with negligible footprints.