CoHadoop++: A load balanced data co-location in Hadoop Distributed File System

Hadoop [4] is a popular distributed computing framework, widely used for business data analytics. While Hadoop performs parallel operations on large data sets, it does not co-locate related data by default. However, performance of log processing operations such as indexing, grouping joins and sessionization on Hadoop can be significantly improved if related logs are partitioned and processed as a group. In order to facilitate this, these partitions need to be placed on the same set of nodes in the cluster. To enable this, a grouping key can be used to identify the related logs. CoHadoop [1], an extension of Hadoop, uses this key to co-locate all those files which correspond to the same key. However, it selects the data nodes randomly for every new key. While doing so, the load introduced on every data node of the cluster is not considered. This leads to an imbalance in the cluster. In order to address this issue, a load balanced co-location algorithm is incorporated into CoHadoop and it is termed as CoHadoop++. It achieves load balance in the cluster through optimal selection of data nodes based their load. Experimental results on CoHadoop++ show that the data placement happens in a load balanced manner.