Distributed Computing with Heterogeneous Servers

Distributed computing is known for its high efficiency of processing large amounts of data in parallel, at the expense of communication load between different servers. Coding was introduced to minimize the communication load by exploiting the repetitive computing, thus drawing great attention within the academia. Most existing works assume that all servers are identical in computational capability, which is inconsistent with practical scenarios. In this paper, we investigate a distributed computing system that consists of two types of servers, i.e., fast servers and slow servers. Due to the heterogeneous computational capabilities within the system, the overall computation time will be delayed by the slow servers, which is called the straggling effect. To this end, we develop a novel framework of coding-based distributed computing to alleviate the straggling effect. Specifically, for a given number of fast servers and slow servers with their corresponding computational capabilities, we aim to minimize the overall computation time by assigning different amounts of workloads to different servers. Further, we derive the information-theoretic lower bound of the communication load of the system, which is shown to be within a constant multiplicative gap to the achievable communication load by our scheme.