Concurrent Hybrid Breadth-First-Search on Distributed PowerGraph for Skewed Graphs

Large-scale graph-structured computation is becoming increasingly important for various data analytics applications. However, most distributed graph processing frameworks do not directly support efficient implementation of sophisticated algorithms requiring massive graph traversals. In this paper, we propose concurrent hybrid breadth-first-search (BFS) algorithm on a popular distributed graph processing framework (Power-Graph and its optimized version PowerLyra). It leverages the small-world property of skewed graphs which apply to most realworld data sets. Hybrid BFS algorithm changes graph traversal direction to save computational workload and message transmissions. Concurrent BFS algorithm enables running multiple BFS simultaneously while sharing vertex explorations efficiently with bit operations. Extensive experiments are conducted to evaluate performance and scalability on a multi-core computer and a cluster of AWS instances. Concurrent hybrid BFS algorithm dramatically increases graph traversal efficiency with respect to the increasing number of concurrent BFS traversals. Compared with PowerGraph, PowerLyra uses fewer resources, while achieving significantly higher performance and better scalability.