The Benefits of Estimated Global Information in DHT Load Balancing

Distributed hash tables (DHT) often rely on uniform hashing for balancing the load among their nodes. However, the most overloaded node may still have a load up to O(log N) times higher than the average load. DHTs with support for range queries cannot rely on hashing to fairly balance the system's load since hashing destroys the order of the stored items. Ensuring a fair load distribution is vital to avoid individual nodes becoming overloaded, potentially leading to node crashes or an incentive not to participate in the system. In both scenarios explicit load balancing schemes can help to spread the load more evenly. In this paper, we improve on existing algorithms for item-based active load balancing by relying on approximations of global properties. We show that the algorithms can be made more efficient by incorporating estimates of properties such as the average load and the standard deviation. Our algorithms reduce the network traffic induced by load balancing while achieving a better load balance than standard algorithms. We also show that these improvements can be applied to passive load balancing algorithms. Compared to DHTs without explicit load balancing, both variants are able to reduce the total maintenance traffic, i.e. item movements due to churn and load balancing, by up to 18%. Simultaneously the system is being balanced achieving a better load distribution.