GraphDuo: A Dual-Model Graph Processing Framework

Algorithms for large-scale natural graph processing can be categorized into two types based on their value propagation behaviors: the unidirectional value propagation algorithms and the bidirectional value propagation algorithms. The graph computation in different types exhibits different properties on how vertices interact with their neighbors and has different requirements on system design. Current distributed graph processing systems usually try to support both types in one general-purpose computing model, which can result in suboptimial performance, high communication overhead, and high resource consumption. In this paper, we propose GraphDuo, a new Spark-based graph processing framework that provides two efficient computing models for unidirectional and bidirectional value propagation algorithms, respectively. Combined with a degree-based graph partitioning scheme, a locality-aware graph layout, and other optimization techniques, GraphDuo achieves low computation cost, low communication overhead, small memory footprint, and good communication balance for two types of algorithms. According to the experimental results, GraphDuo can outperform GraphX from 1.76× to 6.56× and from 1.11× to 1.26× for two types of algorithms, respectively, with much less memory consumption and communication cost. The source code of GraphDuo is publicly available from http://prof.ict.ac.cn/GraphDuo.