Recursive Multi-Section on the Fly: Shared-Memory Streaming Algorithms for Hierarchical Graph Partitioning and Process Mapping

Partitioning a graph into balanced blocks such that few edges run between blocks is a key problem for large-scale distributed processing. In this work, we present a shared-memory streaming multi-recursive partitioning scheme that performs re-cursive multi-sections on the fly without knowing the overall input graph to compute hierarchical partitionings. If a hierarchy is not specified as an input, our approach can also be used as a tool to solve the standard graph partitioning problem. Our approach has a considerably lower running time complexity in comparison with state-of-the-art non-buffered one-pass partitioning algorithms designed for the non-hierarchical graph partitioning case. More-over, if the topology of a distributed system is known, it is possible to further optimize the communication costs by mapping partitions onto processing elements. Our experiments indicate that our algorithm is both faster and produces better process mappings than competing tools. In case of graph partitioning, our framework is up to two orders of magnitude faster at the cost of 5 % more cut edges compared to Fennel.