Invasive Compute Balancing for Applications with Hybrid Parallelization

Achieving high scalability for dynamical adaptive algorithms in HPC is a non-trivial task. In this contex, explicit data migration is typically applied which comes with several challenges such as criteria when and where to migrate data or additional bandwidth requirements. In our work, we use an alternative to data-migration, namely compute-migration on cache-coherent memory systems. We focus on migrating computational power for simulations with dynamical adaptive grids with invasive paradigm. An extension to our existing core-distribution scheduler is presented, distributing the cores depending on the requirements specified by each parallel program instance. In addition, a programming pattern for the hybrid parallel application code is presented to overcome deadlock issues created by applying invasive computing. We validate our approach with benchmark computations for a simulation with artificial workload and furthermore for a realistic scenario based on dynamical adaptive shallow water simulations. Those benchmarks are conducted within a hybrid parallelization environment on a 40-core HPC node. On this shared-memory system, our invasive approach results in faster execution times and higher hardware utilization than the non-invasive approach. Our approach can be applied in general to similar classes of simulations on dynamical adaptive grids.