Designing Virtualization-Aware and Automatic Topology Detection Schemes for Accelerating Hadoop on SR-IOV-Enabled Clouds

Hadoop is gaining more and more popularity in virtualized environments because of the flexibility and elasticity offered by cloud-based systems. Hadoop supports topology-awareness through topology-aware designs in all of its major components. However, there exists no service that can automatically detect the underlying network topology in a scalable and efficient manner, and provide this information to the Hadoop framework. Moreover, the topology-aware designs in Hadoop are not optimized for virtualized platforms. In this paper, we propose a new library called Hadoop-Virt, based on RDMA-Hadoop, which provides with an automatic topology detection module and virtualization-aware designs in Hadoop to fully take the advantage of virtualized environments. Our experimental evaluations show that Hadoop-Virt delivers upto 34% better performance in the default execution mode and upto 52.6% better performance in the distributed mode as compared to default RDMA-Hadoop for SR-IOV-enabled virtualized clusters.