Towards the Dynamic Load Balancing on Heterogeneous Multi-GPU Systems

The advent of multicore systems, joined to the potential acceleration of the graphics processing units, alleviates some well known important architectural problems at the expense of a considerable increment of the programmability wall. The heterogeneity, both at architectural and programming level at the same time, raises the programming difficulties. Adapting existing code and libraries to such systems is a fundamental problem. The performance of this code is affected by the large interdependence between the code and the parallel architecture. We have developed a dynamic load balancing library that allows parallel code to be adapted to a wide variety of heterogeneous systems. The overhead introduced by our system is minimal and the cost to the programmer negligible. This system has been applied to solve load imbalance problems appearing in homogeneous and heterogeneous multi-GPU platforms. As case studies, we consider matrix multiply and resource allocation problems, in different heterogeneous scenarios in multi-GPU systems. The unbalanced nature of these algorithms and heterogeneous systems allowed us to test the success of our load balancing approach.