Performability Comparison of Lustre and HDFS for MR Applications

With its simple principles to achieve parallelism and fault tolerance, the Map-reduce framework has captured wide attention, from traditional high performance computing to marketing organizations. The most popular open source implementation of this framework is Hadoop. Today, the Hadoop stack comprises of various software components including the Hadoop Distributed File System (HDFS), the distributed storage layer amongst others such as GPFS and WASB. The traditional high performance computing has always been at the forefront of developing and deploying cutting edge technology and solutions such as Lustre, a Parallel IO file systems, to meet its ever growing need. To support new and upcoming use cases, there is a focus on tighter integration of Hadoop with existing HPC stacks. In this paper, we share our work on one such integration by analyzing an FSI workload built using map reduce framework and evaluating the performance and reliability of the application on an integrated stack with Hadoop and Lustre through Hadoop extensions such as Hadoop Adapter for Lustre (HAL) and HPC Adapter for MapReduce (HAM) developed by Intel, while comparing the performance against the Hadoop Distributed File System (HDFS). We also carried out perform ability analysis of both the systems, where HDFS ensures reliability using replication factor and Lustre does not replicate any data but ensures reliability by having multiple OSSs connecting to multiple OSTs. The environment used for this evaluation is a 16 nodes HDDP cluster hosted in the Intel Big Data Lab in Swindon (UK). The cluster was divided into two clusters. One 8 node cluster was set up with CDH 5.0.2 and HDFS and another 8 node was set up with CDH 5.0.2 connected to Lustre through Intel HAL. We use Intel Enteprise Edition for Lustre 2.0 for the experiment based on Lustre 2.5. The Lustre setup includes 1 Meta Data Server (MDS) with 1 Meta Data Target (MDT) and 1 Management Target (MGT) and 4 Object Storage Servers (OSSs) with 16 Object Storage Targets (OSTs). Both the systems were evaluated on performance metric 'average query response time' for FSI workload. The data is generated based on FSI application schema while MR jobs are written for few functionalities/queries of the FSI application which are used for the evaluation exercise. Apart from single query execution, both the systems were evaluated for concurrent workload as well. Tests were run for application data volumes varying from 100 GB to 7 TB. From our experiments, with appropriate tuning of Lustre file system, we observe that MR applications on Lustre platform perform at least twice better than that on HDFS. We conducted perform ability analysis of both the systems using Markov Reward Model. We propose linear extrapolation for estimating average query execution time for states exhibiting failure for some nodes and calculated the perform ability with reward for working states as the average query execution time. We assume that the time to failure, detect failure, and repair of both compute nodes as well data nodes are exponentially distributed, and took reasonable parameter values for the same. From our analysis, Expected query execution time for MR applications on Lustre file platform is at least half that of the applications on HDFS platform.