HOOVER: Leveraging OpenSHMEM for High Performance, Flexible Streaming Graph Applications

As the adoption of streaming graph applications increases in the finance, defense, social, and other industries so does the size, velocity, irregularity, and complexity of these streaming graphs. Most existing frameworks for processing streaming, dynamic graphs are too limited in scalability or functionality to efficiently handle these increasingly massive graphs. Many frameworks are either built for shared memory platforms only (limiting the size of the graph that can be stored) or for distributed platforms, but run on slow, high overhead, interpreted, and bulk synchronous platforms. This paper introduces HOOVER, a high performance streaming graph modeling and analysis framework built from scratch to scale on high performance systems and extremely dynamic graphs. HOOVER offers similar APIs to previous streaming graph frameworks, but sits on top of a high performance runtime system designed for modern supercomputers. HOOVER leverages an eventually-consistent consistency model to improve scalability, and offers a number of unique features to users. On micro-benchmarks, HOOVER is shown to be comparable or faster than existing high performance and distributed graph frameworks. Using mini-apps, we also show that HOOVER easily scales to 2,048 PEs on more realistic applications.