Nonconvex Multitask Learning Over Networks

Distributed learning paradigms, such as federated or decentralized learning, have emerged as effective tools in endowing multi -agent systems with collective intelligence, while preserving privacy, robustness to node or link failure, and communication efficiency. While the performance gains of distributed architectures are well established in homogenous environments where agents observe data following similar distributions and aim to solve similar tasks, the same cannot be said for heterogeneous, multitask, or non-iid settings. These heterogeneous environments emerge naturally across a variety of applications and are known to introduce significant performance degradation to classical strategies for distributed learning. Multitask learning, on the other hand, allows a collection of agents to collaborate on solving different, albeit related tasks, without enforcing full consensus and homogeneity. Performance guarantees for decentralized multitask learning algorithms so far have focused on convex objectives, such as those encountered when solving linear or logistic regression objectives, but do not allow for nonconvexity, such as in the case of deep neural networks. We examine the nonconvex learning dynamics of a recently proposed bias-corrected algorithm for nonconvex subspace constrained optimization and demonstrate the effectiveness of the framework in deep multitask learning.