A-MapCG: An Adaptive MapReduce Framework for GPUs

The MapReduce framework proposed by Google to process large data sets is an efficient framework used in many areas, such as social network, scientific research, electronic business, etc. Hence, many MapReduce frameworks are proposed and implemented on different platforms. However, these MapReduce frameworks have limitations, and they cannot handle the collision problem in the map phase, and the unbalanced workload problem in the reduce phase. In this paper, an Adaptive MapReduce Framework (A-MapCG) is proposed based on the MapCG framework, to further improve the MapReduce performance on GPU platforms. Based on the experiments, we observed that for certain MapReduce applications emitting multiple Key/value (K/V) pairs for the same key, the atomic collision problem degrades the map phase performance of the MapReduce framework substantially. In addition, the workload unbalance problem wastes parallel computing resources and limits the overall reduction phase performance of the MapReduce framework on GPU platforms. A-MapCG uses segmentation table and intra-warp combination to reduce the number of collisions during the map phase. A-MapCG also adopts balanced workload assignment to improve the reduce phase performance. The proposed A-MapCG framework is evaluated on the Tesla K40 GPU hosted by Intel Core i7-4790. The case study shows that the map phase of A-MapCG achieves a speedup of 4.63 over MapCG for the test case, Word Count, with a 64MB workload. The average reduce phase speedup of A-MapCG over MapCG with parallel reductions of Word Count is 6.92. The average reduce phase speedup of A-MapCG over MapCG with serial reductions of Word Count is 4.11.