Performance evaluation and optimization of join operation in spark for big data processing

Now Spark is the most important big data distributed computing framework. Join appears frequently in Spark programs, such as PageRank algorithm. Due to join operation will bring lots of Shuffle, thus it costs much time in big data processing. In this paper, we optimize each iteration and total application of PageRank algorithm using broadcast, cache and broadcast-cache in three ways respectively. Experiment data show that broadcast, cache can reduce Shuffle effectively, and thus greatly decrease running time of job. The effect is significant.