Communication Efficient Distributed Approximate Newton Method

In this paper, we develop a communication efficient second order distributed Newton-type algorithm. For communication efficiency, we consider a generic class of δ-approximate compressors (Karimireddy et al., 2019), which includes sign-based compression and top-k sparsification. We provide three potential settings where compression can be employed; and provide rate of convergence for smooth objectives. We show that, in the regime where δ is constant, our theoretical convergence rate matches that of a state-of-the-art distributed second order algorithm called DINGO (Crane and Roosta, 2019). This implies that we get the compression for free in this regime. The full paper can be found at https://tinyurl.com/ujnpt4c.