G-Bench: Fair Benchmarking to Support Innovations in Streaming Graph Systems

In this work, we comment on the current practices for benchmarking streaming graph systems. We deduce that present practices can stifle future innovations in storage design, establish wrong precedents, lead to premature optimizations and complexity, and are a recipe for errors. We analyze why such benchmarking errors are prevalent and propose potential solutions. We further deliver a code base, G-Bench, that can be easily integrated with prior and future research prototypes to provide fair performance comparison metrics without compromising innovations. Our evaluations contradict results published in some recent works.