Communication-Efficient Coded Distributed Multi - Task Learning

Consider a distributed multi-task learning (MTL) framework where the distributed users first train their own models based on the local data and then send the local updates to the server, and the server sends back helpful information by which each user can learn its task independently. Compared to the single-task case, the distributed MTL suffers more severely from the communication bottleneck, because the users wish to learn multiple models, causing the downlink communication load to linearly increase with the total number of tasks. In this paper, we propose a novel scheme named coded distributed multi-task learning, to reduce the communication loads both in the uplink and downlink. The key idea is to exploit the local information stored at the users during the local training, and utilize a particular repetitive placement and computation on the publicly shared dataset such that coded multicasting opportunities can be created at the server and users. Our method for the first time applies coding strategy to reduce the communication cost for distributed MTL framework. Experiments on real-world dataset show that the proposed scheme can substantially reduce the communication load compared to the traditional uncoded approach.