Doopnet: An emulator for network performance analysis of Hadoop clusters using Docker and Mininet

Hadoop is one of the most important Big Data processing and storage systems. In recent years, a lot of efforts have been put to enhance Hadoop's performance from networking perspectives. However, there are limited tools that can help researchers to verify their networking algorithm design in terms of Hadoop's performance. This paper proposes Doopnet which is a framework and toolset for creating Hadoop clusters in a virtualized environment and for monitoring/analysing of Hadoop's networking characteristics under different network configurations. Doopnet enables users to automatically set up a Hadoop cluster over Docker containers running inside Mininet. The Hadoop traffic is collected inside the containers and virtual switches through network flow monitors. The users can easily modify network topologies or configurations through Mininet, observe the networking behaviour through network flow monitors, and analyse the effects of different network settings on Hadoop's performance. Examples are presented to demonstrate how to setup the Doopnet testbed and analyse Hadoop traffic.

I. Introduction

The development of Social Networking, Mobile Computing, and Internet of Things has generated a large volume of data, which has resulted in creations of various Big Data analytics systems, e.g. Hadoop [1], Spark [2], and Flink [3]. During the computation processes in these systems, data movements occur very frequently between computation nodes in a data centre. An analysis from Facebook shows that, on average, data transmissions take 33% of the whole execution time in MapReduce [4] jobs with reduce phases [5]. Furthermore, some skew effects, e.g. partitioning skew, in MapReduce applications may lead to non-uniformed data distributions between different reducers and consequently prolong the execution times significantly [6].