Into the Storm: Descrying Optimal Configurations Using Genetic Algorithms and Bayesian Optimization

Finding an optimal configuration for the number of worker processes and executors for a Storm topology is imperative for maximizing its performance. However, this process is both tedious and time-consuming due to the vast number of parameters to tune, their intertwined relationship with each other, and the temporal overhead of simply rebalancing a topology with a new set of configuration parameters. Without a thorough understanding of the data, the topology, and the framework itself, this endeavor quickly becomes intractable. In order to facilitate the discovery of these parameters, we explore automatic parameter tuners based on either Bayesian optimization or genetic algorithms. To help guide these optimization algorithms, we incorporate both Storm performance data and JMX profiler information. Utilizing a benchmark suite of Storm topologies encompassing a diverse set of performance characteristics, we show that the genetic algorithm approach in particular can quickly find a parameter configuration that nearly doubles performance compared to a common "rule of thumb" baseline.