Memory bandwidth aware scheduling for SMP cluster nodes

Clusters of SMPs are becoming increasingly common. However, the shared memory design of SMPs and the consequential contention between system processors for access to main memory can limit their efficiency significantly. Moreover, the continuous improvement of modern cluster interconnection technologies leads to the network bandwidth being a significant fraction of the total memory bandwidth of the machine, thus the NIC of an SMP cluster node can also become a major consumer of shared memory bus bandwidth. In this paper we first provide experimental evidence that contention on the shared memory bus can have major impact on the total execution time of processes even when no processor sharing is involved, then present the design and implementation of an informed scheduling algorithm for multiprogrammed workloads, which tries to carefully select processes to be coscheduled so that bus saturation is avoided. The input data needed by our scheduler are acquired dynamically, at run-time, using architecture-specific performance monitoring counters and a modified version of the NIC firmware, with no changes to existing application binaries. Experimental comparison between our scheduler and the standard Linux 2.6 O(1) scheduler shows average system throughput improvements in the range of 5-25%.